[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:54:08 dist-test-slave-f892 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:54:08 dist-test-slave-f892 krb5kdc[395](info): set up 2 sockets
Sep 27 06:54:08 dist-test-slave-f892 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 06:54:09.888268 388 mini_kdc.cc:121] Time spent starting KDC: real 2.328s user 0.006s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 06:54:10.048516 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.160s user 0.000s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260927 06:54:10.282970 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.234s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260927 06:54:10.523981 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.241s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/test-user.keytab.
W20260927 06:54:10.641382 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.117s user 0.000s sys 0.004s
Sep 27 06:54:10 dist-test-slave-f892 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:54:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:54:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:54:10.714540 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34507
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:11.022991 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:11.023329 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:11.023489 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:11.030424 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:54:11.030583 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:11.030695 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:11.030797 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:54:11.030871 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:54:11.040550 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:11.041827 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:11.043123 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 06:54:11.055419 408 server_base.cc:1034] running on GCE node
W20260927 06:54:11.055434 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:11.055434 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:11.055691 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:11.056619 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:11.057997 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:11.059289 408 hybrid_clock.cc:648] HybridClock initialized: now 1790492051059218 us; error 64 us; skew 500 ppm
I20260927 06:54:11.059651 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:11 dist-test-slave-f892 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:11.065675 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:54:11.067248 408 webserver.cc:460] Webserver started at http://127.0.97.62:44019/ using document root <none> and password file <none>
I20260927 06:54:11.067978 408 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:11.068125 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:11.068423 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:11.071271 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "eb52aafe463d4819aa48292c3bba4357"
format_stamp: "Formatted at 2026-09-27 06:54:11 on dist-test-slave-f892"
server_key: "d0b6285fabe7f23fa06cf7a438b2597d"
server_key_iv: "9d517449856f5206499a979273ac48ec"
server_key_version: "encryptionkey@0"
I20260927 06:54:11.071985 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "eb52aafe463d4819aa48292c3bba4357"
format_stamp: "Formatted at 2026-09-27 06:54:11 on dist-test-slave-f892"
server_key: "d0b6285fabe7f23fa06cf7a438b2597d"
server_key_iv: "9d517449856f5206499a979273ac48ec"
server_key_version: "encryptionkey@0"
I20260927 06:54:11.076779 408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 06:54:11.079988 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:11.081238 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 06:54:11.081413 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "eb52aafe463d4819aa48292c3bba4357"
format_stamp: "Formatted at 2026-09-27 06:54:11 on dist-test-slave-f892"
server_key: "d0b6285fabe7f23fa06cf7a438b2597d"
server_key_iv: "9d517449856f5206499a979273ac48ec"
server_key_version: "encryptionkey@0"
I20260927 06:54:11.081591 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:11.100404 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:11.104189 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:11.104476 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:11.123847 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34507
I20260927 06:54:11.123878 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34507 every 8 connection(s)
I20260927 06:54:11.125170 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:54:11.128188 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 408
I20260927 06:54:11.128471 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:54:11.128789 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:11.130391 388 external_mini_cluster.cc:1428] Setting key fa9c027581cdd8158a46dd8e12987357
I20260927 06:54:11.139005 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357: Bootstrap starting.
I20260927 06:54:11.141947 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:11.142925 475 log.cc:824] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:11.146071 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357: No bootstrap required, opened a new log
I20260927 06:54:11.155737 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb52aafe463d4819aa48292c3bba4357" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34507 } }
I20260927 06:54:11.156629 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:11.156750 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb52aafe463d4819aa48292c3bba4357, State: Initialized, Role: FOLLOWER
I20260927 06:54:11.157421 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb52aafe463d4819aa48292c3bba4357" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34507 } }
I20260927 06:54:11.157682 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:54:11.157850 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:54:11.158074 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:11.160104 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb52aafe463d4819aa48292c3bba4357" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34507 } }
I20260927 06:54:11.160568 475 leader_election.cc:302] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb52aafe463d4819aa48292c3bba4357; no voters:
I20260927 06:54:11.161132 475 leader_election.cc:288] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:54:11.161324 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357 [term 1 FOLLOWER]: Leader election won for term 1
Sep 27 06:54:11 dist-test-slave-f892 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:11.162617 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357 [term 1 LEADER]: Becoming Leader. State: Replica: eb52aafe463d4819aa48292c3bba4357, State: Running, Role: LEADER
I20260927 06:54:11.163281 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:54:11.163247 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb52aafe463d4819aa48292c3bba4357" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34507 } }
I20260927 06:54:11.167073 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb52aafe463d4819aa48292c3bba4357" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb52aafe463d4819aa48292c3bba4357" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34507 } } }
I20260927 06:54:11.167448 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:11.172168 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb52aafe463d4819aa48292c3bba4357. Latest consensus state: current_term: 1 leader_uuid: "eb52aafe463d4819aa48292c3bba4357" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb52aafe463d4819aa48292c3bba4357" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34507 } } }
I20260927 06:54:11.173290 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:11.175141 487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:54:11.177685 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:11.133974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34582 (local address 127.0.97.62:34507)
0927 06:54:11.135317 (+ 1343us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:11.135333 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:11.139530 (+ 4197us) server_negotiation.cc:407] Connection header received
0927 06:54:11.148415 (+ 8885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:11.148442 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:11.148934 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:11.149337 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:11.152030 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:11.154175 (+ 2145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:11.155836 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:11.157264 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:11.157451 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:11.163948 (+ 6497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:11.163958 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:11.163976 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:11.164022 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:11.170574 (+ 6552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:11.171846 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:11.171858 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:11.171870 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:11.172135 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:11.172786 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:11.172795 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:11.172800 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:11.173655 (+ 855us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:11.174034 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:11.175402 (+ 1368us) server_negotiation.cc:299] Negotiation successful
0927 06:54:11.175818 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1071,"thread_start_us":828,"threads_started":1}
I20260927 06:54:11.185700 487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:54:11.194766 487 catalog_manager.cc:1347] Generated new cluster ID: eb7444bf468c4856b03fbd17afdf8148
I20260927 06:54:11.194864 487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:54:11.231073 487 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:54:11.232371 487 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:54:11.242159 487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb52aafe463d4819aa48292c3bba4357: Generated new TSK 0
I20260927 06:54:11.242944 487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 06:54:11.297144 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34507
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:11.641033 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:11.641278 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:11.641367 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:11.648183 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:11.648273 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:11.648378 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:54:11.657508 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34507
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:11.658737 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:11.659976 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:11.672858 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:11.672868 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:11.672945 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:11.673442 501 server_base.cc:1034] running on GCE node
I20260927 06:54:11.674160 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:11.675572 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:11.676788 501 hybrid_clock.cc:648] HybridClock initialized: now 1790492051676748 us; error 56 us; skew 500 ppm
I20260927 06:54:11.677093 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:11 dist-test-slave-f892 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:11.680253 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:54:11.681609 501 webserver.cc:460] Webserver started at http://127.0.97.1:37083/ using document root <none> and password file <none>
I20260927 06:54:11.682329 501 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:11.682458 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:11.682765 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:11.685606 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "25ffb20e2bcf410b93f4be1cd59a2190"
format_stamp: "Formatted at 2026-09-27 06:54:11 on dist-test-slave-f892"
server_key: "eb4d422b69f7a72dc5fa553e51f7dd78"
server_key_iv: "1614ddcb4b334cd367081beb2041d2cd"
server_key_version: "encryptionkey@0"
I20260927 06:54:11.686539 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "25ffb20e2bcf410b93f4be1cd59a2190"
format_stamp: "Formatted at 2026-09-27 06:54:11 on dist-test-slave-f892"
server_key: "eb4d422b69f7a72dc5fa553e51f7dd78"
server_key_iv: "1614ddcb4b334cd367081beb2041d2cd"
server_key_version: "encryptionkey@0"
I20260927 06:54:11.691195 501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 06:54:11.694926 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:11.696398 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 06:54:11.696574 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "25ffb20e2bcf410b93f4be1cd59a2190"
format_stamp: "Formatted at 2026-09-27 06:54:11 on dist-test-slave-f892"
server_key: "eb4d422b69f7a72dc5fa553e51f7dd78"
server_key_iv: "1614ddcb4b334cd367081beb2041d2cd"
server_key_version: "encryptionkey@0"
I20260927 06:54:11.696817 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:11.717577 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:11.721390 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:11.721796 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:11.722783 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:11.724063 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:11.724175 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:11.724334 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:11.724404 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:11.760407 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42351
I20260927 06:54:11.760439 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42351 every 8 connection(s)
I20260927 06:54:11.761864 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:54:11.765475 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 501
I20260927 06:54:11.765695 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:54:11.766426 388 external_mini_cluster.cc:1428] Setting key c167680143dd8d07efd07f147bddf752
Sep 27 06:54:11 dist-test-slave-f892 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 06:54:11.785418 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:11.764596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52783 (local address 127.0.97.62:34507)
0927 06:54:11.765176 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:11.765183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:11.766341 (+ 1158us) server_negotiation.cc:407] Connection header received
0927 06:54:11.768573 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:11.768579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:11.768664 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:11.768780 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:11.770943 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:11.772642 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:11.774425 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:11.774947 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:11.775102 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:11.779069 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:11.779075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:11.779078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:11.779112 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:11.781875 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:11.782767 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:11.782773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:11.782776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:11.782923 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:11.783432 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:11.783437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:11.783441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:11.783781 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:11.783959 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:11.784790 (+ 831us) server_negotiation.cc:299] Negotiation successful
0927 06:54:11.785033 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":202,"threads_started":1}
I20260927 06:54:11.786729 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:11.765250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34507 (local address 127.0.97.1:52783)
0927 06:54:11.766157 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:11.766214 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:11.768293 (+ 2079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:11.768976 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:11.768984 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:11.769958 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:11.770753 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:11.770778 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:11.772812 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:11.772823 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:11.774271 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:11.774283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:11.775195 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:11.775204 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:11.775481 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:11.776334 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:11.776356 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:11.778803 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:11.782036 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:11.782047 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:11.782065 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:11.782645 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:11.783063 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:11.783070 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:11.783074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:11.783300 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:11.783940 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:11.783948 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:11.784319 (+ 371us) client_negotiation.cc:769] Sending connection context
0927 06:54:11.784660 (+ 341us) client_negotiation.cc:240] Negotiation successful
0927 06:54:11.785315 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":381,"threads_started":1}
I20260927 06:54:11.789229 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34507
I20260927 06:54:11.789557 629 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:11.790385 629 heartbeater.cc:511] Master 127.0.97.62:34507 requested a full tablet report, sending...
I20260927 06:54:11.792965 440 ts_manager.cc:194] Registered new tserver with Master: 25ffb20e2bcf410b93f4be1cd59a2190 (127.0.97.1:42351)
I20260927 06:54:11.795468 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 06:54:11.808883 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34507
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:12.142988 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:12.143240 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:12.143417 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:12.150625 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:12.150713 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:12.150887 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:54:12.161049 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34507
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:12.162561 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:12.163851 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:12.176754 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:12.176885 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:12.177294 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:12.177762 635 server_base.cc:1034] running on GCE node
I20260927 06:54:12.178380 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:12.179184 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:12.180371 635 hybrid_clock.cc:648] HybridClock initialized: now 1790492052180326 us; error 40 us; skew 500 ppm
I20260927 06:54:12.180619 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:12 dist-test-slave-f892 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:12.183696 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:54:12.184996 635 webserver.cc:460] Webserver started at http://127.0.97.2:33151/ using document root <none> and password file <none>
I20260927 06:54:12.185655 635 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:12.185763 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:12.186108 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:12.188721 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "73a1f8a2220b403795a3ec56f267ac98"
format_stamp: "Formatted at 2026-09-27 06:54:12 on dist-test-slave-f892"
server_key: "a6274945a36db48b7a6e583cacf45a66"
server_key_iv: "d5ebfeb4e53fe4d215c2189463b95c7e"
server_key_version: "encryptionkey@0"
I20260927 06:54:12.189395 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "73a1f8a2220b403795a3ec56f267ac98"
format_stamp: "Formatted at 2026-09-27 06:54:12 on dist-test-slave-f892"
server_key: "a6274945a36db48b7a6e583cacf45a66"
server_key_iv: "d5ebfeb4e53fe4d215c2189463b95c7e"
server_key_version: "encryptionkey@0"
I20260927 06:54:12.194572 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 06:54:12.197649 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:12.199282 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:54:12.199452 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "73a1f8a2220b403795a3ec56f267ac98"
format_stamp: "Formatted at 2026-09-27 06:54:12 on dist-test-slave-f892"
server_key: "a6274945a36db48b7a6e583cacf45a66"
server_key_iv: "d5ebfeb4e53fe4d215c2189463b95c7e"
server_key_version: "encryptionkey@0"
I20260927 06:54:12.199651 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:12.217854 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:12.222362 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:12.222661 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:12.223681 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:12.224996 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:12.225100 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:12.225255 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:12.225328 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:12.259660 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34615
I20260927 06:54:12.260115 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34615 every 8 connection(s)
I20260927 06:54:12.260941 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:54:12.264072 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 635
I20260927 06:54:12.264269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:54:12.264830 388 external_mini_cluster.cc:1428] Setting key 8c0d636f89479ea15044721686de704c
Sep 27 06:54:12 dist-test-slave-f892 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 06:54:12.281961 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:12.263389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51795 (local address 127.0.97.62:34507)
0927 06:54:12.263600 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:12.263604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:12.265016 (+ 1412us) server_negotiation.cc:407] Connection header received
0927 06:54:12.267261 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:12.267265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:12.267321 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:12.267396 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:12.269464 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:12.270240 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:12.272359 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:12.272802 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:12.272940 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:12.276501 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:12.276506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:12.276509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:12.276557 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:12.279124 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:12.279790 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:12.279797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:12.279803 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:12.279929 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:12.280359 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:12.280365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:12.280369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:12.280620 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:12.280770 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:12.281426 (+ 656us) server_negotiation.cc:299] Negotiation successful
0927 06:54:12.281631 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 06:54:12.282893 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:12.264025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34507 (local address 127.0.97.2:51795)
0927 06:54:12.264844 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:12.264907 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:12.266981 (+ 2074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:12.267650 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:12.267664 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:12.268469 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:12.269295 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:12.269325 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:12.270391 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:12.270399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:12.272216 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:12.272227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:12.273026 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:12.273032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:12.273247 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:12.273914 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:12.273939 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:12.276289 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:12.279262 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:12.279270 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:12.279285 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:12.279670 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:12.280060 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:12.280067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:12.280071 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:12.280247 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:12.280746 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:12.280753 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:12.281096 (+ 343us) client_negotiation.cc:769] Sending connection context
0927 06:54:12.281354 (+ 258us) client_negotiation.cc:240] Negotiation successful
0927 06:54:12.281817 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":336,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 06:54:12.284240 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34507
I20260927 06:54:12.284534 763 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:12.285372 763 heartbeater.cc:511] Master 127.0.97.62:34507 requested a full tablet report, sending...
I20260927 06:54:12.286837 440 ts_manager.cc:194] Registered new tserver with Master: 73a1f8a2220b403795a3ec56f267ac98 (127.0.97.2:34615)
I20260927 06:54:12.287732 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 06:54:12.306107 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34507
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:12.617244 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:12.617515 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:12.617619 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:12.624346 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:12.624446 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:12.624557 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:54:12.633863 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34507
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:12.635030 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:12.636154 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:12.648432 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:12.648519 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:12.648715 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:12.649871 768 server_base.cc:1034] running on GCE node
I20260927 06:54:12.650501 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:12.651376 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:12.652618 768 hybrid_clock.cc:648] HybridClock initialized: now 1790492052652568 us; error 45 us; skew 500 ppm
I20260927 06:54:12.652915 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:12 dist-test-slave-f892 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:12.655970 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:54:12.657425 768 webserver.cc:460] Webserver started at http://127.0.97.3:41107/ using document root <none> and password file <none>
I20260927 06:54:12.658144 768 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:12.658296 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:12.658612 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:12.661346 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "da0835fb859b4ae6a3dae3b609abaa2f"
format_stamp: "Formatted at 2026-09-27 06:54:12 on dist-test-slave-f892"
server_key: "827cb562afd724bbf7f5f65416dc7172"
server_key_iv: "a22f33b90a5536d1a6e2e2da19965cd8"
server_key_version: "encryptionkey@0"
I20260927 06:54:12.662194 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "da0835fb859b4ae6a3dae3b609abaa2f"
format_stamp: "Formatted at 2026-09-27 06:54:12 on dist-test-slave-f892"
server_key: "827cb562afd724bbf7f5f65416dc7172"
server_key_iv: "a22f33b90a5536d1a6e2e2da19965cd8"
server_key_version: "encryptionkey@0"
I20260927 06:54:12.667017 768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 06:54:12.670706 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:12.671960 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 06:54:12.672129 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "da0835fb859b4ae6a3dae3b609abaa2f"
format_stamp: "Formatted at 2026-09-27 06:54:12 on dist-test-slave-f892"
server_key: "827cb562afd724bbf7f5f65416dc7172"
server_key_iv: "a22f33b90a5536d1a6e2e2da19965cd8"
server_key_version: "encryptionkey@0"
I20260927 06:54:12.672312 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:12.686844 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:12.690814 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:12.691083 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:12.691912 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:12.693162 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:12.693265 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:12.693418 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:12.693495 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:12.729580 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45103
I20260927 06:54:12.729622 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45103 every 8 connection(s)
I20260927 06:54:12.730937 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:54:12.740303 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 768
I20260927 06:54:12.740482 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:54:12.740998 388 external_mini_cluster.cc:1428] Setting key a8569f4885fd0e91dddfdc7e3cf65b58
Sep 27 06:54:12 dist-test-slave-f892 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:12.751075 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:12.733637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40005 (local address 127.0.97.62:34507)
0927 06:54:12.733825 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:12.733829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:12.734930 (+ 1101us) server_negotiation.cc:407] Connection header received
0927 06:54:12.736729 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:12.736734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:12.736824 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:12.736962 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:12.738493 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:12.739225 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:12.741223 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:12.741629 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:12.741782 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:12.746081 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:12.746086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:12.746089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:12.746120 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:12.748347 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:12.748978 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:12.748982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:12.748984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:12.749104 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:12.749608 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:12.749611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:12.749613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:12.749847 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:12.749949 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:12.750687 (+ 738us) server_negotiation.cc:299] Negotiation successful
0927 06:54:12.750845 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 06:54:12.752048 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:12.734106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34507 (local address 127.0.97.3:40005)
0927 06:54:12.734763 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:12.734799 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:12.736453 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:12.737196 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:12.737204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:12.737673 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:12.738301 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:12.738338 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:12.739371 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:12.739377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:12.741065 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:12.741080 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:12.741852 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:12.741857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:12.742177 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:12.742951 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:12.742980 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:12.745856 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:12.748471 (+ 2615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:12.748477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:12.748488 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:12.748867 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:12.749278 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:12.749282 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:12.749285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:12.749472 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:12.749958 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:12.749964 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:12.750335 (+ 371us) client_negotiation.cc:769] Sending connection context
0927 06:54:12.750592 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 06:54:12.750957 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":253,"threads_started":1}
I20260927 06:54:12.753444 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34507
I20260927 06:54:12.753746 896 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:12.754478 896 heartbeater.cc:511] Master 127.0.97.62:34507 requested a full tablet report, sending...
I20260927 06:54:12.755923 440 ts_manager.cc:194] Registered new tserver with Master: da0835fb859b4ae6a3dae3b609abaa2f (127.0.97.3:45103)
I20260927 06:54:12.756992 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40005
I20260927 06:54:12.757359 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 06:54:12 dist-test-slave-f892 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492052, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 06:54:12 dist-test-slave-f892 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492052, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:54:12.798668 629 heartbeater.cc:503] Master 127.0.97.62:34507 was elected leader, sending a full tablet report...
I20260927 06:54:12.801939 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:12.786707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42470 (local address 127.0.97.1:42351)
0927 06:54:12.787354 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:12.787363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:12.787393 (+ 30us) server_negotiation.cc:407] Connection header received
0927 06:54:12.787568 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:12.787576 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:12.787789 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:12.787949 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:12.789649 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:12.790714 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:12.791915 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:12.792442 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:12.792633 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:12.796338 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:12.796348 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:12.796361 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:12.796411 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:12.799401 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:12.800053 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:12.800062 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:12.800067 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:12.800246 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:12.800653 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:12.800656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:12.800658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:12.800949 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:12.801144 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:12.801414 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 06:54:12.801578 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":349,"threads_started":1}
W20260927 06:54:12.803143 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42470
Sep 27 06:54:12 dist-test-slave-f892 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 06:54:12 dist-test-slave-f892 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:54:12.843745 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:12.831360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.0.97.1:42351)
0927 06:54:12.831611 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:12.831620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:12.831744 (+ 124us) server_negotiation.cc:407] Connection header received
0927 06:54:12.831884 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:12.831887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:12.831974 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:12.832062 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:12.833293 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:12.833948 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:12.835103 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:12.835480 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:12.835584 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:12.839132 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:12.839137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:12.839140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:12.839170 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:12.841450 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:12.841973 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:12.841977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:12.841979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:12.842237 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:12.842552 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:12.842556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:12.842559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:12.842841 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:12.843036 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:12.843263 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 06:54:12.843407 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 06:54:12.845141 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 501
I20260927 06:54:12.856952 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:12.845952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42496 (local address 127.0.97.1:42351)
0927 06:54:12.846136 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:12.846140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:12.846524 (+ 384us) server_negotiation.cc:407] Connection header received
0927 06:54:12.846627 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:12.846630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:12.846676 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:12.846746 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:12.847775 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:12.848454 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:12.849602 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:12.849980 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:12.850170 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:12.852099 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:12.852105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:12.852109 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:12.852150 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:12.854988 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:12.855503 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:12.855507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:12.855509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:12.855617 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:12.855903 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:12.855906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:12.855908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:12.856145 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:12.856311 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:12.856572 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 06:54:12.856708 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 06:54:12.857597 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:12.942631 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 501
I20260927 06:54:12.958446 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 635
Sep 27 06:54:12 dist-test-slave-f892 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:54:12.973272 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:12.959306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49318 (local address 127.0.97.2:34615)
0927 06:54:12.959849 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:12.959856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:12.959880 (+ 24us) server_negotiation.cc:407] Connection header received
0927 06:54:12.959935 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:12.959941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:12.960147 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:12.960269 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:12.961315 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:12.962348 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:12.963598 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:12.964115 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:12.964207 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:12.967999 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:12.968008 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:12.968021 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:12.968049 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:12.970883 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:12.971409 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:12.971415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:12.971417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:12.971524 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:12.971831 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:12.971834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:12.971835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:12.972248 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:12.972494 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:12.972719 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 06:54:12.972890 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":302,"threads_started":1}
I20260927 06:54:12.974556 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:13.055349 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 635
I20260927 06:54:13.077865 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 768
Sep 27 06:54:13 dist-test-slave-f892 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 06:54:13.093181 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:13.078779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56534 (local address 127.0.97.3:45103)
0927 06:54:13.079394 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:13.079401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:13.079428 (+ 27us) server_negotiation.cc:407] Connection header received
0927 06:54:13.079483 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:13.079488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:13.079712 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:13.079920 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:13.081064 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:13.082110 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:13.083355 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:13.083916 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:13.084054 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:13.087948 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:13.087957 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:13.087972 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:13.088001 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:13.090611 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:13.091243 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:13.091250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:13.091252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:13.091359 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:13.091690 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:13.091695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:13.091698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:13.091985 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:13.092284 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:13.092603 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 06:54:13.092857 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":321,"threads_started":1}
I20260927 06:54:13.094506 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:13.177143 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 768
I20260927 06:54:13.192451 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 408
I20260927 06:54:13.193589 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:13.290985 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 408
2026-09-27T06:54:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5782 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:54:13 dist-test-slave-f892 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:54:13 dist-test-slave-f892 krb5kdc[917](info): set up 2 sockets
Sep 27 06:54:13 dist-test-slave-f892 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 06:54:14.353915 388 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:54:14 dist-test-slave-f892 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:54:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:54:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:54:14.521936 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36201
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:14.822348 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:14.822681 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:14.822840 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:14.829742 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:54:14.829880 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:14.829990 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:14.830109 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:54:14.830194 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:54:14.839406 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:14.840615 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:14.841840 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:14.853170 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:14.853382 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:14.853365 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:14.854491 930 server_base.cc:1034] running on GCE node
I20260927 06:54:14.855224 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:14.856606 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:14.857846 930 hybrid_clock.cc:648] HybridClock initialized: now 1790492054857799 us; error 45 us; skew 500 ppm
I20260927 06:54:14.858268 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:14 dist-test-slave-f892 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:14.861433 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:54:14.863025 930 webserver.cc:460] Webserver started at http://127.0.97.62:39409/ using document root <none> and password file <none>
I20260927 06:54:14.863668 930 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:14.863763 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:14.863991 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:14.866905 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "8b79263cf619450484a135ed457a081a"
format_stamp: "Formatted at 2026-09-27 06:54:14 on dist-test-slave-f892"
server_key: "77ee4a227f43b81131dddad6c63e5001"
server_key_iv: "d8d64a4d7b621aaf96f692b9121afb67"
server_key_version: "encryptionkey@0"
I20260927 06:54:14.867628 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "8b79263cf619450484a135ed457a081a"
format_stamp: "Formatted at 2026-09-27 06:54:14 on dist-test-slave-f892"
server_key: "77ee4a227f43b81131dddad6c63e5001"
server_key_iv: "d8d64a4d7b621aaf96f692b9121afb67"
server_key_version: "encryptionkey@0"
I20260927 06:54:14.872676 930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 06:54:14.876266 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:14.878057 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:54:14.878238 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "8b79263cf619450484a135ed457a081a"
format_stamp: "Formatted at 2026-09-27 06:54:14 on dist-test-slave-f892"
server_key: "77ee4a227f43b81131dddad6c63e5001"
server_key_iv: "d8d64a4d7b621aaf96f692b9121afb67"
server_key_version: "encryptionkey@0"
I20260927 06:54:14.878451 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:14.913874 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:14.917677 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:14.917898 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:14.935818 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36201
I20260927 06:54:14.935835 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36201 every 8 connection(s)
I20260927 06:54:14.937201 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:54:14.940847 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:14.946106 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 930
I20260927 06:54:14.946326 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:54:14.946941 388 external_mini_cluster.cc:1428] Setting key 5dc460085569923b1bf7f0fcec147a2b
I20260927 06:54:14.952490 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a: Bootstrap starting.
I20260927 06:54:14.956050 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:14.957206 997 log.cc:824] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:14.959993 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a: No bootstrap required, opened a new log
Sep 27 06:54:14 dist-test-slave-f892 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:14.969314 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b79263cf619450484a135ed457a081a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36201 } }
I20260927 06:54:14.969702 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:14.969796 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b79263cf619450484a135ed457a081a, State: Initialized, Role: FOLLOWER
I20260927 06:54:14.970470 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b79263cf619450484a135ed457a081a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36201 } }
I20260927 06:54:14.970733 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:54:14.970759 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:14.949031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55966 (local address 127.0.97.62:36201)
0927 06:54:14.949939 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:14.949960 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:14.950024 (+ 64us) server_negotiation.cc:407] Connection header received
0927 06:54:14.952355 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:14.952381 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:14.952872 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:14.953262 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:14.954596 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:14.956631 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:14.958102 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:14.958971 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:14.959137 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:14.962894 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:14.962904 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:14.962927 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:14.962980 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:14.966764 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:14.967388 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:14.967399 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:14.967407 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:14.967599 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:14.967971 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:14.967976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:14.967979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:14.968656 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:14.968907 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:14.969269 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 06:54:14.969661 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":620,"thread_start_us":387,"threads_started":1}
I20260927 06:54:14.970916 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:54:14.971134 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:14.973776 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b79263cf619450484a135ed457a081a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36201 } }
I20260927 06:54:14.974258 997 leader_election.cc:302] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b79263cf619450484a135ed457a081a; no voters:
I20260927 06:54:14.974848 997 leader_election.cc:288] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:54:14.974997 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:14.976289 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a [term 1 LEADER]: Becoming Leader. State: Replica: 8b79263cf619450484a135ed457a081a, State: Running, Role: LEADER
I20260927 06:54:14.976886 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b79263cf619450484a135ed457a081a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36201 } }
I20260927 06:54:14.977048 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:54:14.979295 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b79263cf619450484a135ed457a081a. Latest consensus state: current_term: 1 leader_uuid: "8b79263cf619450484a135ed457a081a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b79263cf619450484a135ed457a081a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36201 } } }
I20260927 06:54:14.979612 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:14.980091 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b79263cf619450484a135ed457a081a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b79263cf619450484a135ed457a081a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36201 } } }
I20260927 06:54:14.980412 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:14.991369 1009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:54:14.996323 1009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:54:15.004339 1009 catalog_manager.cc:1347] Generated new cluster ID: 7d46aa6043d543ae9c6c2edd0377f3bf
I20260927 06:54:15.004443 1009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:54:15.024869 1009 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:54:15.026947 1009 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:54:15.035405 1009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b79263cf619450484a135ed457a081a: Generated new TSK 0
I20260927 06:54:15.036408 1009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 06:54:15.090453 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36201
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:15.408327 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:15.408625 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:15.408821 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:15.416430 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:15.416596 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:15.416790 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:54:15.426626 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36201
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:15.427908 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:15.429215 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:15.442032 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:15.442133 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:15.441992 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:15.442777 1023 server_base.cc:1034] running on GCE node
I20260927 06:54:15.443367 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:15.444275 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:15.445498 1023 hybrid_clock.cc:648] HybridClock initialized: now 1790492055445452 us; error 61 us; skew 500 ppm
I20260927 06:54:15.445793 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:15 dist-test-slave-f892 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:15.448698 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:54:15.450073 1023 webserver.cc:460] Webserver started at http://127.0.97.1:39577/ using document root <none> and password file <none>
I20260927 06:54:15.450686 1023 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:15.450824 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:15.451119 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:15.453788 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "2682944838d547ec93e1435852a4d386"
format_stamp: "Formatted at 2026-09-27 06:54:15 on dist-test-slave-f892"
server_key: "a5193d5319ced86f297bcc916124ce8d"
server_key_iv: "371bf1f7f02a7a8988977806e33e0297"
server_key_version: "encryptionkey@0"
I20260927 06:54:15.454602 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2682944838d547ec93e1435852a4d386"
format_stamp: "Formatted at 2026-09-27 06:54:15 on dist-test-slave-f892"
server_key: "a5193d5319ced86f297bcc916124ce8d"
server_key_iv: "371bf1f7f02a7a8988977806e33e0297"
server_key_version: "encryptionkey@0"
I20260927 06:54:15.460145 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 06:54:15.463658 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:15.465051 1023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 06:54:15.465240 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "2682944838d547ec93e1435852a4d386"
format_stamp: "Formatted at 2026-09-27 06:54:15 on dist-test-slave-f892"
server_key: "a5193d5319ced86f297bcc916124ce8d"
server_key_iv: "371bf1f7f02a7a8988977806e33e0297"
server_key_version: "encryptionkey@0"
I20260927 06:54:15.465451 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:15.479998 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:15.484215 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:15.484483 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:15.485317 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:15.486582 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:15.486650 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:15.486810 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:15.486881 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:15.518680 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41305
I20260927 06:54:15.518766 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41305 every 8 connection(s)
I20260927 06:54:15.519924 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:54:15.527400 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 1023
I20260927 06:54:15.527616 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:54:15.528319 388 external_mini_cluster.cc:1428] Setting key 8f33177933e4f2450351e6bb4b0ee4a7
Sep 27 06:54:15 dist-test-slave-f892 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:15.541314 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:15.522453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43203 (local address 127.0.97.62:36201)
0927 06:54:15.522866 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:15.522872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:15.523922 (+ 1050us) server_negotiation.cc:407] Connection header received
0927 06:54:15.525613 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:15.525617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:15.525678 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:15.525772 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:15.527229 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:15.528360 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:15.530482 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:15.531061 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:15.531224 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:15.535002 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:15.535009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:15.535021 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:15.535081 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:15.538051 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:15.538980 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:15.538983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:15.538985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:15.539085 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:15.539574 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:15.539577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:15.539578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:15.539854 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:15.539962 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:15.540910 (+ 948us) server_negotiation.cc:299] Negotiation successful
0927 06:54:15.541067 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":196,"threads_started":1}
I20260927 06:54:15.542927 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:15.523045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36201 (local address 127.0.97.1:43203)
0927 06:54:15.523764 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:15.523801 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:15.525352 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:15.525977 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:15.525985 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:15.526508 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:15.527061 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:15.527081 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:15.528536 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:15.528540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:15.530350 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:15.530361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:15.531302 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:15.531307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:15.531587 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:15.532267 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:15.532288 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:15.534746 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:15.538189 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:15.538200 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:15.538220 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:15.538854 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:15.539192 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:15.539199 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:15.539205 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:15.539462 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:15.539986 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:15.539996 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:15.540448 (+ 452us) client_negotiation.cc:769] Sending connection context
0927 06:54:15.540781 (+ 333us) client_negotiation.cc:240] Negotiation successful
0927 06:54:15.541337 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":289,"threads_started":1}
I20260927 06:54:15.544523 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36201
I20260927 06:54:15.544840 1151 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 06:54:15.545624 1151 heartbeater.cc:511] Master 127.0.97.62:36201 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 06:54:15.547947 962 ts_manager.cc:194] Registered new tserver with Master: 2682944838d547ec93e1435852a4d386 (127.0.97.1:41305)
I20260927 06:54:15.549355 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 06:54:15.569326 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36201
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:15.872885 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:15.873136 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:15.873333 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:15.880595 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:15.880694 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:15.880839 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:54:15.889832 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36201
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:15.890956 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:15.892117 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:15.904609 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:15.904562 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:15.904567 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:15.905283 1157 server_base.cc:1034] running on GCE node
I20260927 06:54:15.905791 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:15.906724 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:15.907925 1157 hybrid_clock.cc:648] HybridClock initialized: now 1790492055907890 us; error 28 us; skew 500 ppm
I20260927 06:54:15.908241 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:15 dist-test-slave-f892 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:15.911242 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:54:15.912585 1157 webserver.cc:460] Webserver started at http://127.0.97.2:43341/ using document root <none> and password file <none>
I20260927 06:54:15.913264 1157 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:15.913394 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:15.913714 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:15.916488 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "447b0145b0a14676a80052314ba09e2f"
format_stamp: "Formatted at 2026-09-27 06:54:15 on dist-test-slave-f892"
server_key: "8472c479274a66674411d494ed90926d"
server_key_iv: "6fa1e8d72710e1b45bd1425dd22017a8"
server_key_version: "encryptionkey@0"
I20260927 06:54:15.917227 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "447b0145b0a14676a80052314ba09e2f"
format_stamp: "Formatted at 2026-09-27 06:54:15 on dist-test-slave-f892"
server_key: "8472c479274a66674411d494ed90926d"
server_key_iv: "6fa1e8d72710e1b45bd1425dd22017a8"
server_key_version: "encryptionkey@0"
I20260927 06:54:15.921890 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 06:54:15.925102 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:15.926353 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:54:15.926514 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "447b0145b0a14676a80052314ba09e2f"
format_stamp: "Formatted at 2026-09-27 06:54:15 on dist-test-slave-f892"
server_key: "8472c479274a66674411d494ed90926d"
server_key_iv: "6fa1e8d72710e1b45bd1425dd22017a8"
server_key_version: "encryptionkey@0"
I20260927 06:54:15.926700 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:15.963356 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:15.967161 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:15.967447 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:15.968341 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:15.969679 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:15.969785 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:15.969937 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:15.970008 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:16.002229 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36601
I20260927 06:54:16.002274 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36601 every 8 connection(s)
I20260927 06:54:16.003423 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:54:16.005710 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 1157
I20260927 06:54:16.005895 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:54:16.006397 388 external_mini_cluster.cc:1428] Setting key ae58ee530d604c4d6e3bfebec7bab847
Sep 27 06:54:16 dist-test-slave-f892 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 06:54:16.025156 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.005917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56003 (local address 127.0.97.62:36201)
0927 06:54:16.006125 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:16.006134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:16.007260 (+ 1126us) server_negotiation.cc:407] Connection header received
0927 06:54:16.009382 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:16.009386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:16.009443 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:16.009520 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:16.011530 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.012208 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.014041 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.014420 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.014514 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.018533 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:16.018542 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:16.018550 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:16.018592 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:16.022212 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:16.022908 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:16.022914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:16.022918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:16.023074 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:16.023476 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:16.023484 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:16.023490 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:16.023855 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:16.024012 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:16.024698 (+ 686us) server_negotiation.cc:299] Negotiation successful
0927 06:54:16.024866 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 06:54:16.026291 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.006380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36201 (local address 127.0.97.2:56003)
0927 06:54:16.007106 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:16.007141 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:16.009128 (+ 1987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:16.009786 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:16.009800 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:16.010573 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:16.011363 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:16.011389 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.012357 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.012365 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:16.013873 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:16.013889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.014641 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.014650 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:16.014973 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.015664 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:16.015695 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:16.018315 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:16.022351 (+ 4036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:16.022357 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:16.022369 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:16.022769 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:16.023184 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:16.023187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:16.023189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:16.023339 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:16.023971 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:16.023976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:16.024352 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 06:54:16.024590 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 06:54:16.024958 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":297,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 06:54:16.027695 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36201
I20260927 06:54:16.027981 1285 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:16.028649 1285 heartbeater.cc:511] Master 127.0.97.62:36201 requested a full tablet report, sending...
I20260927 06:54:16.030129 962 ts_manager.cc:194] Registered new tserver with Master: 447b0145b0a14676a80052314ba09e2f (127.0.97.2:36601)
I20260927 06:54:16.031134 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 06:54:16.047601 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36201
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:16.350118 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:16.350369 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:16.350459 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:16.357216 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:16.357318 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:16.357419 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:54:16.366305 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36201
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:16.367461 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:16.368773 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:16.381687 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:16.381636 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:16.381637 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:16.382691 1290 server_base.cc:1034] running on GCE node
I20260927 06:54:16.383311 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:16.384342 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:16.385581 1290 hybrid_clock.cc:648] HybridClock initialized: now 1790492056385543 us; error 25 us; skew 500 ppm
I20260927 06:54:16.385890 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:16 dist-test-slave-f892 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:16.393016 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:54:16.394501 1290 webserver.cc:460] Webserver started at http://127.0.97.3:38907/ using document root <none> and password file <none>
I20260927 06:54:16.395182 1290 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:16.395337 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:16.395648 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:16.398499 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "f692acfd302a48a193c75e77d9a932ff"
format_stamp: "Formatted at 2026-09-27 06:54:16 on dist-test-slave-f892"
server_key: "c70d8db23457fb90b749aea8223c01ea"
server_key_iv: "b472c19cc79078a1af781f5fc79c3bae"
server_key_version: "encryptionkey@0"
I20260927 06:54:16.399252 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f692acfd302a48a193c75e77d9a932ff"
format_stamp: "Formatted at 2026-09-27 06:54:16 on dist-test-slave-f892"
server_key: "c70d8db23457fb90b749aea8223c01ea"
server_key_iv: "b472c19cc79078a1af781f5fc79c3bae"
server_key_version: "encryptionkey@0"
I20260927 06:54:16.404073 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 06:54:16.407274 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:16.408424 1290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260927 06:54:16.408600 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "f692acfd302a48a193c75e77d9a932ff"
format_stamp: "Formatted at 2026-09-27 06:54:16 on dist-test-slave-f892"
server_key: "c70d8db23457fb90b749aea8223c01ea"
server_key_iv: "b472c19cc79078a1af781f5fc79c3bae"
server_key_version: "encryptionkey@0"
I20260927 06:54:16.408795 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:16.424467 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:16.428345 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:16.428650 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:16.429632 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:16.431089 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:16.431187 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:16.431334 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:16.431407 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:16.463775 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45097
I20260927 06:54:16.463821 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45097 every 8 connection(s)
I20260927 06:54:16.465075 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:54:16.472184 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 1290
I20260927 06:54:16.472366 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:54:16.472944 388 external_mini_cluster.cc:1428] Setting key ed27a7981e7dd1ba9d63848208162bc0
Sep 27 06:54:16 dist-test-slave-f892 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:16.484911 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.467591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53839 (local address 127.0.97.62:36201)
0927 06:54:16.467765 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:16.467769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:16.469002 (+ 1233us) server_negotiation.cc:407] Connection header received
0927 06:54:16.470900 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:16.470905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:16.470966 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:16.471051 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:16.472500 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.473559 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.475728 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.476208 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.476303 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.479549 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:16.479554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:16.479557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:16.479588 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:16.482108 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:16.482951 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:16.482955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:16.482957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:16.483122 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:16.483503 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:16.483506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:16.483508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:16.483848 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:16.483961 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:16.484534 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 06:54:16.484679 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 06:54:16.485821 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.468090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36201 (local address 127.0.97.3:53839)
0927 06:54:16.468830 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:16.468873 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:16.470668 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:16.471251 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:16.471259 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:16.471735 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:16.472304 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:16.472332 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.473729 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.473734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:16.475599 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:16.475616 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.476390 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.476394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:16.476586 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.477222 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:16.477241 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:16.479377 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:16.482254 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:16.482260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:16.482271 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:16.482833 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:16.483221 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:16.483224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:16.483227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:16.483405 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:16.483952 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:16.483958 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:16.484234 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 06:54:16.484440 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 06:54:16.484776 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":257,"threads_started":1}
I20260927 06:54:16.487353 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36201
I20260927 06:54:16.487672 1418 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:16.488417 1418 heartbeater.cc:511] Master 127.0.97.62:36201 requested a full tablet report, sending...
I20260927 06:54:16.489650 961 ts_manager.cc:194] Registered new tserver with Master: f692acfd302a48a193c75e77d9a932ff (127.0.97.3:45097)
I20260927 06:54:16.490486 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53839
I20260927 06:54:16.500221 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:54:16.518798 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.507002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.0.97.62:36201)
0927 06:54:16.507169 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:16.507173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:16.507681 (+ 508us) server_negotiation.cc:407] Connection header received
0927 06:54:16.507810 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:16.507813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:16.507870 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:16.507951 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:16.509367 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.510139 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.511299 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.511688 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.511853 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.514186 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:16.514191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:16.514194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:16.514223 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:16.516506 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:16.517053 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:16.517056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:16.517058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:16.517164 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:16.517444 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:16.517447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:16.517449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:16.517754 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:16.517913 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:16.518389 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 06:54:16.518538 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 06:54:16.525561 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:54:16.532120 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:54:16.552438 1151 heartbeater.cc:503] Master 127.0.97.62:36201 was elected leader, sending a full tablet report...
I20260927 06:54:16.557051 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.543693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44494 (local address 127.0.97.1:41305)
0927 06:54:16.545689 (+ 1996us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:16.545702 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:16.546678 (+ 976us) server_negotiation.cc:407] Connection header received
0927 06:54:16.547077 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:16.547085 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:16.547349 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:16.547543 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:16.549273 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.550640 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.554508 (+ 3868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.555938 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.556100 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.556326 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:16.556585 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 06:54:16.556735 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1766,"thread_start_us":1603,"threads_started":1}
I20260927 06:54:16.557255 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.543528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41305 (local address 127.0.0.1:44494)
0927 06:54:16.546535 (+ 3007us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:16.546570 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:16.546842 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:16.547899 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:16.547910 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:16.547949 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:16.549095 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:16.549103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.550763 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.550771 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:16.554347 (+ 3576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:16.554358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.556140 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.556145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:16.556348 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.556409 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 06:54:16.556544 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 06:54:16.556725 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2778,"mutex_wait_us":381,"spinlock_wait_cycles":4480,"thread_start_us":2623,"threads_started":1}
I20260927 06:54:16.557425 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.547077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45097 (local address 127.0.0.1:37498)
0927 06:54:16.549266 (+ 2189us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:16.549282 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:16.549394 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:16.549958 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:16.549962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:16.549986 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:16.550320 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:16.550330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.552112 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.552120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:16.554691 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:16.554705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.556657 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.556663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:16.556809 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.556840 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 06:54:16.556930 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 06:54:16.557045 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2070,"mutex_wait_us":47,"thread_start_us":1959,"threads_started":1}
I20260927 06:54:16.557659 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.544160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36601 (local address 127.0.0.1:42910)
0927 06:54:16.546620 (+ 2460us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:16.546643 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:16.546842 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:16.548171 (+ 1329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:16.548176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:16.548202 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:16.548658 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:16.548673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.550848 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.550857 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:16.554345 (+ 3488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:16.554358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.556326 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.556332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:16.556491 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.556524 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 06:54:16.556625 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 06:54:16.556757 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2265,"mutex_wait_us":209,"spinlock_wait_cycles":3968,"thread_start_us":2577,"threads_started":1}
I20260927 06:54:16.557705 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.543968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42910 (local address 127.0.97.2:36601)
0927 06:54:16.547649 (+ 3681us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:16.547657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:16.547681 (+ 24us) server_negotiation.cc:407] Connection header received
0927 06:54:16.547763 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:16.547770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:16.547975 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:16.548271 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:16.549244 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.550726 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.554527 (+ 3801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.555864 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.555984 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.556182 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:16.557224 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0927 06:54:16.557384 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3522,"thread_start_us":3116,"threads_started":1}
I20260927 06:54:16.558279 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.547718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.0.97.3:45097)
0927 06:54:16.548900 (+ 1182us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:16.548906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:16.549387 (+ 481us) server_negotiation.cc:407] Connection header received
0927 06:54:16.549548 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:16.549555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:16.549750 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:16.549942 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:16.550486 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.551981 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.554866 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.556186 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.556315 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.556506 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:16.557704 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0927 06:54:16.557989 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":914,"threads_started":1}
I20260927 06:54:16.561187 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4ae21612d44bb396f7af744104cfdb (DEFAULT_TABLE table=test-table [id=1d344412562b47718ac2554e2361ab7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:16.561236 1353 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4ae21612d44bb396f7af744104cfdb (DEFAULT_TABLE table=test-table [id=1d344412562b47718ac2554e2361ab7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:16.561232 1220 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4ae21612d44bb396f7af744104cfdb (DEFAULT_TABLE table=test-table [id=1d344412562b47718ac2554e2361ab7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:16.562331 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4ae21612d44bb396f7af744104cfdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:16.562511 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4ae21612d44bb396f7af744104cfdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:16.562525 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4ae21612d44bb396f7af744104cfdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:16.574447 1436 tablet_bootstrap.cc:492] T 1b4ae21612d44bb396f7af744104cfdb P 2682944838d547ec93e1435852a4d386: Bootstrap starting.
I20260927 06:54:16.577948 1436 tablet_bootstrap.cc:654] T 1b4ae21612d44bb396f7af744104cfdb P 2682944838d547ec93e1435852a4d386: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:16.579046 1436 log.cc:824] T 1b4ae21612d44bb396f7af744104cfdb P 2682944838d547ec93e1435852a4d386: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:16.580416 1437 tablet_bootstrap.cc:492] T 1b4ae21612d44bb396f7af744104cfdb P 447b0145b0a14676a80052314ba09e2f: Bootstrap starting.
I20260927 06:54:16.582638 1436 tablet_bootstrap.cc:492] T 1b4ae21612d44bb396f7af744104cfdb P 2682944838d547ec93e1435852a4d386: No bootstrap required, opened a new log
I20260927 06:54:16.582806 1438 tablet_bootstrap.cc:492] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff: Bootstrap starting.
I20260927 06:54:16.583055 1436 ts_tablet_manager.cc:1397] T 1b4ae21612d44bb396f7af744104cfdb P 2682944838d547ec93e1435852a4d386: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260927 06:54:16.585765 1437 tablet_bootstrap.cc:654] T 1b4ae21612d44bb396f7af744104cfdb P 447b0145b0a14676a80052314ba09e2f: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:16.586923 1437 log.cc:824] T 1b4ae21612d44bb396f7af744104cfdb P 447b0145b0a14676a80052314ba09e2f: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:16.588590 1438 tablet_bootstrap.cc:654] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:16.589836 1437 tablet_bootstrap.cc:492] T 1b4ae21612d44bb396f7af744104cfdb P 447b0145b0a14676a80052314ba09e2f: No bootstrap required, opened a new log
I20260927 06:54:16.590144 1437 ts_tablet_manager.cc:1397] T 1b4ae21612d44bb396f7af744104cfdb P 447b0145b0a14676a80052314ba09e2f: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260927 06:54:16.590145 1438 log.cc:824] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:16.592634 1438 tablet_bootstrap.cc:492] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff: No bootstrap required, opened a new log
I20260927 06:54:16.592928 1438 ts_tablet_manager.cc:1397] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260927 06:54:16.593107 1436 raft_consensus.cc:348] T 1b4ae21612d44bb396f7af744104cfdb P 2682944838d547ec93e1435852a4d386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2682944838d547ec93e1435852a4d386" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41305 } } peers { permanent_uuid: "447b0145b0a14676a80052314ba09e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36601 } } peers { permanent_uuid: "f692acfd302a48a193c75e77d9a932ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45097 } }
I20260927 06:54:16.593509 1436 raft_consensus.cc:374] T 1b4ae21612d44bb396f7af744104cfdb P 2682944838d547ec93e1435852a4d386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:16.593643 1436 raft_consensus.cc:729] T 1b4ae21612d44bb396f7af744104cfdb P 2682944838d547ec93e1435852a4d386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2682944838d547ec93e1435852a4d386, State: Initialized, Role: FOLLOWER
I20260927 06:54:16.594264 1436 consensus_queue.cc:260] T 1b4ae21612d44bb396f7af744104cfdb P 2682944838d547ec93e1435852a4d386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2682944838d547ec93e1435852a4d386" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41305 } } peers { permanent_uuid: "447b0145b0a14676a80052314ba09e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36601 } } peers { permanent_uuid: "f692acfd302a48a193c75e77d9a932ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45097 } }
I20260927 06:54:16.599261 1436 ts_tablet_manager.cc:1428] T 1b4ae21612d44bb396f7af744104cfdb P 2682944838d547ec93e1435852a4d386: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260927 06:54:16.600968 1437 raft_consensus.cc:348] T 1b4ae21612d44bb396f7af744104cfdb P 447b0145b0a14676a80052314ba09e2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2682944838d547ec93e1435852a4d386" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41305 } } peers { permanent_uuid: "447b0145b0a14676a80052314ba09e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36601 } } peers { permanent_uuid: "f692acfd302a48a193c75e77d9a932ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45097 } }
I20260927 06:54:16.601378 1437 raft_consensus.cc:374] T 1b4ae21612d44bb396f7af744104cfdb P 447b0145b0a14676a80052314ba09e2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:16.601475 1437 raft_consensus.cc:729] T 1b4ae21612d44bb396f7af744104cfdb P 447b0145b0a14676a80052314ba09e2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447b0145b0a14676a80052314ba09e2f, State: Initialized, Role: FOLLOWER
I20260927 06:54:16.601480 1438 raft_consensus.cc:348] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2682944838d547ec93e1435852a4d386" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41305 } } peers { permanent_uuid: "447b0145b0a14676a80052314ba09e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36601 } } peers { permanent_uuid: "f692acfd302a48a193c75e77d9a932ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45097 } }
I20260927 06:54:16.601940 1438 raft_consensus.cc:374] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:16.602116 1438 raft_consensus.cc:729] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f692acfd302a48a193c75e77d9a932ff, State: Initialized, Role: FOLLOWER
I20260927 06:54:16.602098 1437 consensus_queue.cc:260] T 1b4ae21612d44bb396f7af744104cfdb P 447b0145b0a14676a80052314ba09e2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2682944838d547ec93e1435852a4d386" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41305 } } peers { permanent_uuid: "447b0145b0a14676a80052314ba09e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36601 } } peers { permanent_uuid: "f692acfd302a48a193c75e77d9a932ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45097 } }
I20260927 06:54:16.602727 1438 consensus_queue.cc:260] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2682944838d547ec93e1435852a4d386" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41305 } } peers { permanent_uuid: "447b0145b0a14676a80052314ba09e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36601 } } peers { permanent_uuid: "f692acfd302a48a193c75e77d9a932ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45097 } }
I20260927 06:54:16.603401 1285 heartbeater.cc:503] Master 127.0.97.62:36201 was elected leader, sending a full tablet report...
I20260927 06:54:16.603847 1437 ts_tablet_manager.cc:1428] T 1b4ae21612d44bb396f7af744104cfdb P 447b0145b0a14676a80052314ba09e2f: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 06:54:16.604614 1418 heartbeater.cc:503] Master 127.0.97.62:36201 was elected leader, sending a full tablet report...
I20260927 06:54:16.605310 1438 ts_tablet_manager.cc:1428] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff: Time spent starting tablet: real 0.012s user 0.003s sys 0.007s
I20260927 06:54:16.635267 1444 raft_consensus.cc:482] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:54:16.635542 1444 raft_consensus.cc:504] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2682944838d547ec93e1435852a4d386" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41305 } } peers { permanent_uuid: "447b0145b0a14676a80052314ba09e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36601 } } peers { permanent_uuid: "f692acfd302a48a193c75e77d9a932ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45097 } }
I20260927 06:54:16.637699 1444 leader_election.cc:288] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2682944838d547ec93e1435852a4d386 (127.0.97.1:41305), 447b0145b0a14676a80052314ba09e2f (127.0.97.2:36601)
I20260927 06:54:16.643828 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.638192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57577 (local address 127.0.97.1:41305)
0927 06:54:16.638383 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:16.638391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:16.638414 (+ 23us) server_negotiation.cc:407] Connection header received
0927 06:54:16.638487 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:16.638494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:16.638574 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:16.638696 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:16.639245 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.640203 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.641980 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.643059 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.643163 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.643230 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:16.643562 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 06:54:16.643663 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 06:54:16.643918 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.637897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41305 (local address 127.0.97.3:57577)
0927 06:54:16.638180 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:16.638199 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:16.638313 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:16.638730 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:16.638737 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:16.638768 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:16.639057 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:16.639069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.640344 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.640351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:16.641818 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:16.641830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.643207 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.643213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:16.643386 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.643420 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 06:54:16.643517 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 06:54:16.643661 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260927 06:54:16.644897 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.638148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55345 (local address 127.0.97.2:36601)
0927 06:54:16.638385 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:16.638392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:16.639228 (+ 836us) server_negotiation.cc:407] Connection header received
0927 06:54:16.639362 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:16.639367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:16.639436 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:16.639551 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:16.639946 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.640833 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.642827 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.643891 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.644027 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.644128 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:16.644619 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 06:54:16.644725 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 06:54:16.645027 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.638315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36601 (local address 127.0.97.3:55345)
0927 06:54:16.639117 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:16.639138 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:16.639243 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:16.639541 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:16.639544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:16.639559 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:16.639807 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:16.639814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.640982 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.640987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:16.642671 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:16.642688 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.644257 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.644261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:16.644456 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.644487 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 06:54:16.644586 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 06:54:16.644724 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":434,"threads_started":1}
I20260927 06:54:16.644817 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae21612d44bb396f7af744104cfdb" candidate_uuid: "f692acfd302a48a193c75e77d9a932ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2682944838d547ec93e1435852a4d386" is_pre_election: true
I20260927 06:54:16.645352 1106 raft_consensus.cc:2454] T 1b4ae21612d44bb396f7af744104cfdb P 2682944838d547ec93e1435852a4d386 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f692acfd302a48a193c75e77d9a932ff in term 0.
I20260927 06:54:16.645962 1306 leader_election.cc:302] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2682944838d547ec93e1435852a4d386, f692acfd302a48a193c75e77d9a932ff; no voters:
I20260927 06:54:16.645953 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae21612d44bb396f7af744104cfdb" candidate_uuid: "f692acfd302a48a193c75e77d9a932ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447b0145b0a14676a80052314ba09e2f" is_pre_election: true
I20260927 06:54:16.646366 1444 raft_consensus.cc:2781] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:54:16.646514 1444 raft_consensus.cc:482] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:54:16.646454 1240 raft_consensus.cc:2454] T 1b4ae21612d44bb396f7af744104cfdb P 447b0145b0a14676a80052314ba09e2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f692acfd302a48a193c75e77d9a932ff in term 0.
I20260927 06:54:16.646621 1444 raft_consensus.cc:3037] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:16.649297 1444 raft_consensus.cc:504] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2682944838d547ec93e1435852a4d386" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41305 } } peers { permanent_uuid: "447b0145b0a14676a80052314ba09e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36601 } } peers { permanent_uuid: "f692acfd302a48a193c75e77d9a932ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45097 } }
I20260927 06:54:16.649835 1444 leader_election.cc:288] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff [CANDIDATE]: Term 1 election: Requested vote from peers 2682944838d547ec93e1435852a4d386 (127.0.97.1:41305), 447b0145b0a14676a80052314ba09e2f (127.0.97.2:36601)
I20260927 06:54:16.650218 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae21612d44bb396f7af744104cfdb" candidate_uuid: "f692acfd302a48a193c75e77d9a932ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447b0145b0a14676a80052314ba09e2f"
I20260927 06:54:16.650209 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae21612d44bb396f7af744104cfdb" candidate_uuid: "f692acfd302a48a193c75e77d9a932ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2682944838d547ec93e1435852a4d386"
I20260927 06:54:16.650403 1240 raft_consensus.cc:3037] T 1b4ae21612d44bb396f7af744104cfdb P 447b0145b0a14676a80052314ba09e2f [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:16.650401 1106 raft_consensus.cc:3037] T 1b4ae21612d44bb396f7af744104cfdb P 2682944838d547ec93e1435852a4d386 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:16.653481 1106 raft_consensus.cc:2454] T 1b4ae21612d44bb396f7af744104cfdb P 2682944838d547ec93e1435852a4d386 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f692acfd302a48a193c75e77d9a932ff in term 1.
I20260927 06:54:16.653481 1240 raft_consensus.cc:2454] T 1b4ae21612d44bb396f7af744104cfdb P 447b0145b0a14676a80052314ba09e2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f692acfd302a48a193c75e77d9a932ff in term 1.
I20260927 06:54:16.653860 1306 leader_election.cc:302] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2682944838d547ec93e1435852a4d386, 447b0145b0a14676a80052314ba09e2f, f692acfd302a48a193c75e77d9a932ff; no voters:
I20260927 06:54:16.654230 1444 raft_consensus.cc:2781] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:16.654657 1444 raft_consensus.cc:686] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff [term 1 LEADER]: Becoming Leader. State: Replica: f692acfd302a48a193c75e77d9a932ff, State: Running, Role: LEADER
I20260927 06:54:16.655084 1444 consensus_queue.cc:237] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2682944838d547ec93e1435852a4d386" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41305 } } peers { permanent_uuid: "447b0145b0a14676a80052314ba09e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36601 } } peers { permanent_uuid: "f692acfd302a48a193c75e77d9a932ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45097 } }
I20260927 06:54:16.659408 962 catalog_manager.cc:5495] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff reported cstate change: term changed from 0 to 1, leader changed from <none> to f692acfd302a48a193c75e77d9a932ff (127.0.97.3). New cstate: current_term: 1 leader_uuid: "f692acfd302a48a193c75e77d9a932ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2682944838d547ec93e1435852a4d386" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "447b0145b0a14676a80052314ba09e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f692acfd302a48a193c75e77d9a932ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45097 } health_report { overall_health: HEALTHY } } }
Sep 27 06:54:16 dist-test-slave-f892 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:54:16.681447 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.666989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44498 (local address 127.0.97.1:41305)
0927 06:54:16.667215 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:16.667224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:16.668006 (+ 782us) server_negotiation.cc:407] Connection header received
0927 06:54:16.668132 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:16.668136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:16.668195 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:16.668277 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:16.669728 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.670508 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.671678 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.672084 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.672175 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.676065 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:16.676077 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:16.676092 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:16.676128 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:16.678894 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:16.679537 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:16.679546 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:16.679548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:16.679668 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:16.680048 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:16.680055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:16.680063 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:16.680377 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:16.680712 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:16.680916 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 06:54:16.681128 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 06:54:16 dist-test-slave-f892 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492056, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260927 06:54:16.716742 1419 tablet.cc:2369] T 1b4ae21612d44bb396f7af744104cfdb P f692acfd302a48a193c75e77d9a932ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 06:54:16 dist-test-slave-f892 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492056, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:54:16.725167 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.711850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44506 (local address 127.0.97.1:41305)
0927 06:54:16.712086 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:16.712093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:16.712322 (+ 229us) server_negotiation.cc:407] Connection header received
0927 06:54:16.712483 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:16.712487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:16.712550 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:16.712643 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:16.713976 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.714747 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.715917 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.716327 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.716416 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.720387 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:16.720392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:16.720395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:16.720425 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:16.722998 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:16.723548 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:16.723552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:16.723554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:16.723677 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:16.724007 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:16.724010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:16.724012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:16.724234 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:16.724424 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:16.724664 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 06:54:16.724809 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
Sep 27 06:54:16 dist-test-slave-f892 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260927 06:54:16.755347 1286 tablet.cc:2369] T 1b4ae21612d44bb396f7af744104cfdb P 447b0145b0a14676a80052314ba09e2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 06:54:16 dist-test-slave-f892 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:54:16.767499 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.753925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44522 (local address 127.0.97.1:41305)
0927 06:54:16.754209 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:16.754217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:16.754491 (+ 274us) server_negotiation.cc:407] Connection header received
0927 06:54:16.754704 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:16.754708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:16.754768 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:16.754878 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:16.756626 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.757284 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.758579 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.758998 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.759132 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.762717 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:16.762722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:16.762725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:16.762755 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:16.764953 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:16.765570 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:16.765600 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:16.765603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:16.765717 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:16.766157 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:16.766166 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:16.766169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:16.766420 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:16.766690 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:16.766949 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 06:54:16.767177 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 06:54:16.769141 1066 tablet_service.cc:2908] Rejecting scan request for tablet 1b4ae21612d44bb396f7af744104cfdb: Uninitialized: clean time has not yet been initialized
I20260927 06:54:16.771865 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 1023
W20260927 06:54:16.772172 1152 tablet.cc:2369] T 1b4ae21612d44bb396f7af744104cfdb P 2682944838d547ec93e1435852a4d386: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:54:16.785341 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.773620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.0.97.1:41305)
0927 06:54:16.773770 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:16.773774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:16.774150 (+ 376us) server_negotiation.cc:407] Connection header received
0927 06:54:16.774263 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:16.774267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:16.774329 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:16.774402 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:16.775400 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.776083 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.777316 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.777738 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.777873 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.780342 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:16.780348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:16.780352 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:16.780391 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:16.782786 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:16.783455 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:16.783461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:16.783464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:16.783582 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:16.783938 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:16.783944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:16.783947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:16.784216 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:16.784501 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:16.784770 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 06:54:16.784966 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 06:54:16.786214 1146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:16.873643 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 1023
I20260927 06:54:16.892647 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 1157
Sep 27 06:54:16 dist-test-slave-f892 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:54:16.907346 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:16.893312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42926 (local address 127.0.97.2:36601)
0927 06:54:16.893499 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:16.893505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:16.893525 (+ 20us) server_negotiation.cc:407] Connection header received
0927 06:54:16.893728 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:16.893733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:16.893827 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:16.893943 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:16.895098 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.895854 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.897177 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:16.897598 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:16.897739 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:16.901784 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:16.901805 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:16.901821 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:16.901850 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:16.904454 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:16.905111 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:16.905120 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:16.905123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:16.905238 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:16.905638 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:16.905644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:16.905647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:16.905993 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:16.906380 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:16.906821 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 06:54:16.907036 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 06:54:16.907981 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:17.001626 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 1157
I20260927 06:54:17.017704 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 1290
Sep 27 06:54:17 dist-test-slave-f892 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 06:54:17.032393 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:17.018655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.0.97.3:45097)
0927 06:54:17.018828 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:17.018832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:17.018853 (+ 21us) server_negotiation.cc:407] Connection header received
0927 06:54:17.018908 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:17.018912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:17.018971 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:17.019068 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:17.020222 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:17.020928 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:17.022346 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:17.022733 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:17.022833 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:17.026864 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:17.026879 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:17.026894 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:17.026942 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:17.029715 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:17.030452 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:17.030459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:17.030461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:17.030640 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:17.031013 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:17.031017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:17.031019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:17.031404 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:17.031639 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:17.031860 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 06:54:17.032044 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 06:54:17.033082 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:17.120082 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 1290
I20260927 06:54:17.136389 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 930
I20260927 06:54:17.137544 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:17.245764 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 930
2026-09-27T06:54:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3957 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:54:17 dist-test-slave-f892 krb5kdc[1465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:54:17 dist-test-slave-f892 krb5kdc[1465](info): set up 2 sockets
Sep 27 06:54:17 dist-test-slave-f892 krb5kdc[1465](info): commencing operation
krb5kdc: starting...
W20260927 06:54:18.301964 388 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:54:18 dist-test-slave-f892 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:54:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:54:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:54:18.475049 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42255
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:18.774837 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:18.775175 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:18.775327 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:18.782584 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:54:18.782732 1478 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 06:54:18.782830 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:18.782951 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:18.783035 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:54:18.783124 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:54:18.792838 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42255
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:18.794116 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:18.795382 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:18.806495 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:18.806579 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:18.806548 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:18.807238 1478 server_base.cc:1034] running on GCE node
I20260927 06:54:18.807888 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:18.809055 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:18.810293 1478 hybrid_clock.cc:648] HybridClock initialized: now 1790492058810243 us; error 44 us; skew 500 ppm
I20260927 06:54:18.810606 1478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:18 dist-test-slave-f892 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:18.813709 1478 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:54:18.815065 1478 webserver.cc:460] Webserver started at http://127.0.97.62:39017/ using document root <none> and password file <none>
I20260927 06:54:18.815693 1478 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:18.815816 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:18.816131 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:18.819064 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "238c20bc311043d181686d91b29877d6"
format_stamp: "Formatted at 2026-09-27 06:54:18 on dist-test-slave-f892"
server_key: "30c1bd036b41e2aaff0513410d1c047c"
server_key_iv: "21ebc5bc9d2ce33d3ade0fd63a764603"
server_key_version: "encryptionkey@0"
I20260927 06:54:18.819846 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "238c20bc311043d181686d91b29877d6"
format_stamp: "Formatted at 2026-09-27 06:54:18 on dist-test-slave-f892"
server_key: "30c1bd036b41e2aaff0513410d1c047c"
server_key_iv: "21ebc5bc9d2ce33d3ade0fd63a764603"
server_key_version: "encryptionkey@0"
I20260927 06:54:18.824631 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 06:54:18.827831 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:18.829003 1478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 06:54:18.829180 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "238c20bc311043d181686d91b29877d6"
format_stamp: "Formatted at 2026-09-27 06:54:18 on dist-test-slave-f892"
server_key: "30c1bd036b41e2aaff0513410d1c047c"
server_key_iv: "21ebc5bc9d2ce33d3ade0fd63a764603"
server_key_version: "encryptionkey@0"
I20260927 06:54:18.829384 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:18.853701 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:18.857988 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:18.858281 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:18.879813 1478 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42255
I20260927 06:54:18.879855 1554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42255 every 8 connection(s)
I20260927 06:54:18.881656 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:54:18.886396 1555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:18.890020 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 1478
I20260927 06:54:18.890241 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:54:18.891050 388 external_mini_cluster.cc:1428] Setting key 1aeb9729416bc880d52f396b27362e56
I20260927 06:54:18.899453 1555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6: Bootstrap starting.
I20260927 06:54:18.903146 1555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:18.904466 1555 log.cc:824] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6: Log is configured to *not* fsync() on all Append() calls
Sep 27 06:54:18 dist-test-slave-f892 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:18.907405 1555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6: No bootstrap required, opened a new log
I20260927 06:54:18.913435 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:18.892997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57780 (local address 127.0.97.62:42255)
0927 06:54:18.893708 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:18.893719 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:18.893761 (+ 42us) server_negotiation.cc:407] Connection header received
0927 06:54:18.895337 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:18.895358 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:18.895743 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:18.896060 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:18.897813 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:18.899135 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:18.900785 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:18.901460 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:18.901601 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:18.906451 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:18.906458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:18.906472 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:18.906507 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:18.909535 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:18.910327 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:18.910336 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:18.910342 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:18.910491 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:18.910862 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:18.910865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:18.910867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:18.911428 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:18.911636 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:18.912071 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 06:54:18.912438 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":321,"threads_started":1}
I20260927 06:54:18.916461 1555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238c20bc311043d181686d91b29877d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42255 } }
I20260927 06:54:18.916977 1555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:18.917200 1555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 238c20bc311043d181686d91b29877d6, State: Initialized, Role: FOLLOWER
I20260927 06:54:18.917725 1555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238c20bc311043d181686d91b29877d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42255 } }
I20260927 06:54:18.917973 1555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:54:18.918179 1555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:54:18.918344 1555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:18.920545 1555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238c20bc311043d181686d91b29877d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42255 } }
I20260927 06:54:18.920987 1555 leader_election.cc:302] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 238c20bc311043d181686d91b29877d6; no voters:
I20260927 06:54:18.921478 1555 leader_election.cc:288] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:54:18.921694 1560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:18.922425 1560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6 [term 1 LEADER]: Becoming Leader. State: Replica: 238c20bc311043d181686d91b29877d6, State: Running, Role: LEADER
I20260927 06:54:18.922899 1560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238c20bc311043d181686d91b29877d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42255 } }
I20260927 06:54:18.923437 1555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:54:18.926156 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 238c20bc311043d181686d91b29877d6. Latest consensus state: current_term: 1 leader_uuid: "238c20bc311043d181686d91b29877d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238c20bc311043d181686d91b29877d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42255 } } }
I20260927 06:54:18.926192 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "238c20bc311043d181686d91b29877d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238c20bc311043d181686d91b29877d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42255 } } }
I20260927 06:54:18.926522 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:18.926582 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:18.927999 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:54:18.933944 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:54:18.943295 1568 catalog_manager.cc:1347] Generated new cluster ID: 61117d4b74cd4cefa967cd254d54a58d
I20260927 06:54:18.943428 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:54:18.957170 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:54:18.958873 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:54:18.967438 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6: Generated new TSK 0
I20260927 06:54:18.968154 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 06:54:19.019737 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42255
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:19.349400 1581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:19.349651 1581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:19.349797 1581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:19.356829 1581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:19.356914 1581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:19.357096 1581 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 06:54:19.357218 1581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:54:19.366901 1581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:19.368163 1581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:19.369445 1581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:19.381844 1586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:19.381810 1587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:19.381875 1589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:19.383020 1581 server_base.cc:1034] running on GCE node
I20260927 06:54:19.383710 1581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:19.384629 1581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:19.385867 1581 hybrid_clock.cc:648] HybridClock initialized: now 1790492059385812 us; error 35 us; skew 500 ppm
I20260927 06:54:19.386184 1581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:19 dist-test-slave-f892 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:19.392498 1581 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:54:19.393914 1581 webserver.cc:460] Webserver started at http://127.0.97.1:37545/ using document root <none> and password file <none>
I20260927 06:54:19.394634 1581 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:19.394773 1581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:19.395092 1581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:19.397917 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "835f92c8c5d54d1f9d937d1812a78ee4"
format_stamp: "Formatted at 2026-09-27 06:54:19 on dist-test-slave-f892"
server_key: "9de8373657a8536ab2c677e96441796a"
server_key_iv: "f29bfca3ea260df7ad929780ede6494c"
server_key_version: "encryptionkey@0"
I20260927 06:54:19.398751 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "835f92c8c5d54d1f9d937d1812a78ee4"
format_stamp: "Formatted at 2026-09-27 06:54:19 on dist-test-slave-f892"
server_key: "9de8373657a8536ab2c677e96441796a"
server_key_iv: "f29bfca3ea260df7ad929780ede6494c"
server_key_version: "encryptionkey@0"
I20260927 06:54:19.403775 1581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 06:54:19.407123 1596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:19.408385 1581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 06:54:19.408574 1581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "835f92c8c5d54d1f9d937d1812a78ee4"
format_stamp: "Formatted at 2026-09-27 06:54:19 on dist-test-slave-f892"
server_key: "9de8373657a8536ab2c677e96441796a"
server_key_iv: "f29bfca3ea260df7ad929780ede6494c"
server_key_version: "encryptionkey@0"
I20260927 06:54:19.408785 1581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:19.434664 1581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:19.438782 1581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:19.439054 1581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:19.441706 1581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:19.441813 1581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:19.442037 1581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:19.442135 1581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 06:54:19 dist-test-slave-f892 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:19.466408 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:19.442133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58703 (local address 127.0.97.62:42255)
0927 06:54:19.442652 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:19.442661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:19.443864 (+ 1203us) server_negotiation.cc:407] Connection header received
0927 06:54:19.446674 (+ 2810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:19.446679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:19.446739 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:19.446830 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:19.449158 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:19.449880 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:19.453209 (+ 3329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:19.453800 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:19.453948 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:19.458874 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:19.458880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:19.458883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:19.458922 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:19.462286 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:19.463311 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:19.463315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:19.463317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:19.463429 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:19.464039 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:19.464043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:19.464045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:19.464425 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:19.464574 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:19.465584 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0927 06:54:19.465860 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":236,"threads_started":1}
I20260927 06:54:19.467661 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:19.442741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42255 (local address 127.0.97.1:58703)
0927 06:54:19.443661 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:19.443718 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:19.446374 (+ 2656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:19.447128 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:19.447141 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:19.447947 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:19.448929 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:19.448982 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:19.450073 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:19.450082 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:19.452973 (+ 2891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:19.452994 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:19.454109 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:19.454120 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:19.454464 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:19.455283 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:19.455321 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:19.458640 (+ 3319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:19.462486 (+ 3846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:19.462498 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:19.462517 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:19.463122 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:19.463549 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:19.463556 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:19.463564 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:19.463891 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:19.464568 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:19.464576 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:19.465122 (+ 546us) client_negotiation.cc:769] Sending connection context
0927 06:54:19.465453 (+ 331us) client_negotiation.cc:240] Negotiation successful
0927 06:54:19.466177 (+ 724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":368,"threads_started":1}
I20260927 06:54:19.503512 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:19.482101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58737 (local address 127.0.97.62:42255)
0927 06:54:19.482333 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:19.482340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:19.486949 (+ 4609us) server_negotiation.cc:407] Connection header received
0927 06:54:19.487158 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:19.487166 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:19.487234 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:19.487327 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:19.489133 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:19.490501 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:19.492661 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:19.493363 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:19.493476 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:19.496313 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:19.496321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:19.496328 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:19.496373 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:19.500246 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:19.501310 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:19.501317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:19.501324 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:19.501499 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:19.502046 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:19.502053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:19.502058 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:19.502437 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:19.502577 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:19.502948 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 06:54:19.503166 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 06:54:19.503634 1680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:19.482737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42255 (local address 127.0.97.1:58737)
0927 06:54:19.486843 (+ 4106us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:19.486869 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:19.487035 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:19.487358 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:19.487362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:19.488072 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:19.488794 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:19.488813 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:19.490689 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:19.490695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:19.492490 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:19.492504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:19.493787 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:19.493796 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:19.494037 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:19.494777 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:19.494818 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:19.496075 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:19.500432 (+ 4357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:19.500443 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:19.500449 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:19.501135 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:19.501671 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:19.501678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:19.501683 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:19.501857 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:19.502535 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:19.502540 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:19.502720 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 06:54:19.502860 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 06:54:19.503196 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3956,"thread_start_us":3833,"threads_started":1}
I20260927 06:54:19.505728 1581 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38011
I20260927 06:54:19.505738 1716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38011 every 8 connection(s)
I20260927 06:54:19.507583 1581 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:54:19.508975 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 1581
I20260927 06:54:19.509195 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:54:19.509969 388 external_mini_cluster.cc:1428] Setting key b7c21d1c7d82794098ec5dc34e6b5340
I20260927 06:54:19.515646 1717 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42255
I20260927 06:54:19.515869 1717 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:19.516603 1717 heartbeater.cc:511] Master 127.0.97.62:42255 requested a full tablet report, sending...
I20260927 06:54:19.518793 1510 ts_manager.cc:194] Registered new tserver with Master: 835f92c8c5d54d1f9d937d1812a78ee4 (127.0.97.1:38011)
I20260927 06:54:19.521059 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58703
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 06:54:19.557722 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42255
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:19.866569 1721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:19.866835 1721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:19.866930 1721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:19.874033 1721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:19.874125 1721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:19.874212 1721 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 06:54:19.874289 1721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:54:19.883623 1721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:19.884717 1721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:19.885926 1721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:19.902292 1727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:19.902375 1726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:19.902338 1729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:19.903475 1721 server_base.cc:1034] running on GCE node
I20260927 06:54:19.904008 1721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:19.904825 1721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:19.906108 1721 hybrid_clock.cc:648] HybridClock initialized: now 1790492059906047 us; error 43 us; skew 500 ppm
I20260927 06:54:19.906446 1721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:19 dist-test-slave-f892 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:19.909420 1721 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:54:19.910871 1721 webserver.cc:460] Webserver started at http://127.0.97.2:33761/ using document root <none> and password file <none>
I20260927 06:54:19.911532 1721 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:19.911659 1721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:19.911968 1721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:19.914881 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "3f98fcd5b83b4e6ab4e29353bc59204a"
format_stamp: "Formatted at 2026-09-27 06:54:19 on dist-test-slave-f892"
server_key: "fd2dbf1cb3209e19e892d5bd2fe973af"
server_key_iv: "c294a31ebbb0bcbb03dfcfc87ca90b60"
server_key_version: "encryptionkey@0"
I20260927 06:54:19.915709 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3f98fcd5b83b4e6ab4e29353bc59204a"
format_stamp: "Formatted at 2026-09-27 06:54:19 on dist-test-slave-f892"
server_key: "fd2dbf1cb3209e19e892d5bd2fe973af"
server_key_iv: "c294a31ebbb0bcbb03dfcfc87ca90b60"
server_key_version: "encryptionkey@0"
I20260927 06:54:19.920516 1721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 06:54:19.923666 1736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:19.924928 1721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 06:54:19.925092 1721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "3f98fcd5b83b4e6ab4e29353bc59204a"
format_stamp: "Formatted at 2026-09-27 06:54:19 on dist-test-slave-f892"
server_key: "fd2dbf1cb3209e19e892d5bd2fe973af"
server_key_iv: "c294a31ebbb0bcbb03dfcfc87ca90b60"
server_key_version: "encryptionkey@0"
I20260927 06:54:19.925293 1721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:19.947302 1721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:19.951021 1721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:19.951287 1721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:19.953938 1721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:19.954053 1721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 06:54:19.954241 1721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:19.954339 1721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 06:54:19 dist-test-slave-f892 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:19.975482 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:19.954291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38491 (local address 127.0.97.62:42255)
0927 06:54:19.954477 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:19.954482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:19.955950 (+ 1468us) server_negotiation.cc:407] Connection header received
0927 06:54:19.957959 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:19.957966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:19.958104 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:19.958238 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:19.960413 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:19.961780 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:19.964228 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:19.965003 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:19.965178 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:19.969628 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:19.969634 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:19.969639 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:19.969677 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:19.972519 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:19.973257 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:19.973264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:19.973268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:19.973386 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:19.973820 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:19.973827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:19.973830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:19.974173 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:19.974320 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:19.974961 (+ 641us) server_negotiation.cc:299] Negotiation successful
0927 06:54:19.975179 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 06:54:19.976352 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:19.955013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42255 (local address 127.0.97.2:38491)
0927 06:54:19.955797 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:19.955830 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:19.957599 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:19.958538 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:19.958549 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:19.959374 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:19.960211 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:19.960243 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:19.961967 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:19.961975 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:19.964089 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:19.964110 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:19.965287 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:19.965294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:19.965481 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:19.966139 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:19.966175 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:19.969402 (+ 3227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:19.972653 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:19.972661 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:19.972675 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:19.973145 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:19.973488 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:19.973494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:19.973498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:19.973669 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:19.974309 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:19.974316 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:19.974620 (+ 304us) client_negotiation.cc:769] Sending connection context
0927 06:54:19.974869 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 06:54:19.975307 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":345,"threads_started":1}
I20260927 06:54:19.996795 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:19.982065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42071 (local address 127.0.97.62:42255)
0927 06:54:19.982209 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:19.982213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:19.982804 (+ 591us) server_negotiation.cc:407] Connection header received
0927 06:54:19.982997 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:19.983000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:19.983057 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:19.983165 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:19.984576 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:19.985248 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:19.987748 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:19.988308 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:19.988435 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:19.990982 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:19.990989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:19.990995 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:19.991055 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:19.994365 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:19.994998 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:19.995003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:19.995007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:19.995143 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:19.995467 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:19.995471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:19.995475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:19.995760 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:19.995900 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:19.996271 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 06:54:19.996461 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 06:54:19.996876 1799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:19.982100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42255 (local address 127.0.97.2:42071)
0927 06:54:19.982701 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:19.982722 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:19.982862 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:19.983230 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:19.983238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:19.983962 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:19.984418 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:19.984433 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:19.985377 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:19.985385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:19.987581 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:19.987598 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:19.988553 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:19.988565 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:19.988830 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:19.989496 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:19.989526 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:19.990782 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:19.994486 (+ 3704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:19.994492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:19.994496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:19.994880 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:19.995241 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:19.995247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:19.995250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:19.995364 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:19.995880 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:19.995884 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:19.996051 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 06:54:19.996180 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 06:54:19.996458 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":355,"threads_started":1}
I20260927 06:54:20.009616 1721 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35601
I20260927 06:54:20.009637 1855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35601 every 8 connection(s)
I20260927 06:54:20.010859 1721 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:54:20.012754 1856 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42255
I20260927 06:54:20.012959 1856 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:20.013595 1856 heartbeater.cc:511] Master 127.0.97.62:42255 requested a full tablet report, sending...
I20260927 06:54:20.013908 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 1721
I20260927 06:54:20.014142 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:54:20.014735 388 external_mini_cluster.cc:1428] Setting key d7079536990ab433c2b8ff9705c35985
I20260927 06:54:20.015185 1510 ts_manager.cc:194] Registered new tserver with Master: 3f98fcd5b83b4e6ab4e29353bc59204a (127.0.97.2:35601)
I20260927 06:54:20.016193 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38491
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 06:54:20.052904 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42255
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:20.360675 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:20.360935 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:20.361106 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:20.367987 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:20.368080 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:20.368238 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 06:54:20.368335 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:54:20.377748 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:20.378959 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:20.380316 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:20.393085 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:20.393154 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:20.393129 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:20.394184 1860 server_base.cc:1034] running on GCE node
I20260927 06:54:20.394804 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:20.395704 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:20.396929 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790492060396858 us; error 43 us; skew 500 ppm
I20260927 06:54:20.397217 1860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:20 dist-test-slave-f892 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:20.400333 1860 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:54:20.401791 1860 webserver.cc:460] Webserver started at http://127.0.97.3:36005/ using document root <none> and password file <none>
I20260927 06:54:20.402479 1860 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:20.402626 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:20.402928 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:20.405622 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "0e77fecbe105443c94a94a44c2b1f1d0"
format_stamp: "Formatted at 2026-09-27 06:54:20 on dist-test-slave-f892"
server_key: "78834c89367c43df3eb89c49ca85d516"
server_key_iv: "8fe78026917b91ad181fe1ef9bb24391"
server_key_version: "encryptionkey@0"
I20260927 06:54:20.406381 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0e77fecbe105443c94a94a44c2b1f1d0"
format_stamp: "Formatted at 2026-09-27 06:54:20 on dist-test-slave-f892"
server_key: "78834c89367c43df3eb89c49ca85d516"
server_key_iv: "8fe78026917b91ad181fe1ef9bb24391"
server_key_version: "encryptionkey@0"
I20260927 06:54:20.411005 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 06:54:20.414367 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:20.415846 1860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 06:54:20.416018 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "0e77fecbe105443c94a94a44c2b1f1d0"
format_stamp: "Formatted at 2026-09-27 06:54:20 on dist-test-slave-f892"
server_key: "78834c89367c43df3eb89c49ca85d516"
server_key_iv: "8fe78026917b91ad181fe1ef9bb24391"
server_key_version: "encryptionkey@0"
I20260927 06:54:20.416232 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:20.432966 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:20.436820 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:20.437078 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:20.439726 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:20.439803 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:20.440001 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:20.440088 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 06:54:20 dist-test-slave-f892 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:20.465519 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.440270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57777 (local address 127.0.97.62:42255)
0927 06:54:20.440495 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:20.440501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:20.441966 (+ 1465us) server_negotiation.cc:407] Connection header received
0927 06:54:20.445093 (+ 3127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:20.445101 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:20.445196 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:20.445351 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:20.447842 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.449052 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.453173 (+ 4121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.453850 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.454080 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.459352 (+ 5272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:20.459357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:20.459360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:20.459390 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:20.462323 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:20.463098 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:20.463107 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:20.463112 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:20.463288 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:20.463791 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:20.463796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:20.463799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:20.464085 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:20.464245 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:20.464938 (+ 693us) server_negotiation.cc:299] Negotiation successful
0927 06:54:20.465162 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 06:54:20.466910 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.440904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42255 (local address 127.0.97.3:57777)
0927 06:54:20.441766 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:20.441822 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:20.444676 (+ 2854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:20.445614 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:20.445622 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:20.446671 (+ 1049us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:20.447541 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:20.447574 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.449236 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.449245 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:20.452986 (+ 3741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:20.453005 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.454251 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.454260 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:20.454470 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.455404 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:20.455430 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:20.459038 (+ 3608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:20.462503 (+ 3465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:20.462512 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:20.462530 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:20.462965 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:20.463460 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:20.463476 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:20.463480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:20.463662 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:20.464217 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:20.464226 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:20.464570 (+ 344us) client_negotiation.cc:769] Sending connection context
0927 06:54:20.464827 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 06:54:20.465352 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":344,"threads_started":1}
I20260927 06:54:20.485903 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.473813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53209 (local address 127.0.97.62:42255)
0927 06:54:20.474049 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:20.474056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:20.474680 (+ 624us) server_negotiation.cc:407] Connection header received
0927 06:54:20.474856 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:20.474859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:20.474922 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:20.475002 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:20.476383 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.477109 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.478875 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.479377 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.479501 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.481080 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:20.481086 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:20.481091 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:20.481128 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:20.483495 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:20.484137 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:20.484144 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:20.484148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:20.484261 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:20.484599 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:20.484606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:20.484611 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:20.484843 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:20.484995 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:20.485385 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 06:54:20.485584 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 06:54:20.486061 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.473790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42255 (local address 127.0.97.3:53209)
0927 06:54:20.474562 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:20.474586 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:20.474745 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:20.475057 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:20.475064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:20.475784 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:20.476216 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:20.476231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.477318 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.477324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:20.478747 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:20.478758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.479553 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.479560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:20.479714 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.480264 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:20.480279 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:20.480912 (+ 633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:20.483642 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:20.483649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:20.483660 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:20.484028 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:20.484366 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:20.484372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:20.484375 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:20.484498 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:20.484954 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:20.484959 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:20.485148 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 06:54:20.485280 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 06:54:20.485588 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":417,"threads_started":1}
I20260927 06:54:20.501148 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35967
I20260927 06:54:20.501178 1994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35967 every 8 connection(s)
I20260927 06:54:20.502401 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:54:20.504542 1995 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42255
I20260927 06:54:20.504771 1995 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:20.505514 1995 heartbeater.cc:511] Master 127.0.97.62:42255 requested a full tablet report, sending...
I20260927 06:54:20.506855 1510 ts_manager.cc:194] Registered new tserver with Master: 0e77fecbe105443c94a94a44c2b1f1d0 (127.0.97.3:35967)
I20260927 06:54:20.507637 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57777
I20260927 06:54:20.508562 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 1860
I20260927 06:54:20.508733 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:54:20.509212 388 external_mini_cluster.cc:1428] Setting key 52a966a31c5669f51492b663e0afff3c
I20260927 06:54:20.511965 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:54:20.524606 1717 heartbeater.cc:503] Master 127.0.97.62:42255 was elected leader, sending a full tablet report...
Sep 27 06:54:20 dist-test-slave-f892 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492060, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 06:54:20 dist-test-slave-f892 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492060, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:20.555655 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.542395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.0.97.62:42255)
0927 06:54:20.542573 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:20.542577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:20.542821 (+ 244us) server_negotiation.cc:407] Connection header received
0927 06:54:20.543042 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:20.543046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:20.543112 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:20.543209 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:20.544541 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.545345 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.546679 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.547160 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.547281 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.550884 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:20.550889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:20.550893 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:20.550924 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:20.553401 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:20.553951 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:20.553955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:20.553958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:20.554167 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:20.554513 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:20.554517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:20.554519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:20.554768 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:20.554947 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:20.555197 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 06:54:20.555357 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 06:54:20.559485 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:54:20.565943 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:54:20.591032 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.577307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50268 (local address 127.0.97.1:38011)
0927 06:54:20.578679 (+ 1372us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:20.578686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:20.579241 (+ 555us) server_negotiation.cc:407] Connection header received
0927 06:54:20.579635 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:20.579641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:20.579860 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:20.580048 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:20.580946 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.582976 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.586282 (+ 3306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.588190 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.588283 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.588490 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:20.590357 (+ 1867us) server_negotiation.cc:299] Negotiation successful
0927 06:54:20.590604 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1220,"thread_start_us":1086,"threads_started":1}
I20260927 06:54:20.592378 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.577095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38011 (local address 127.0.0.1:50268)
0927 06:54:20.579099 (+ 2004us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:20.579129 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:20.579323 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:20.580319 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:20.580326 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:20.580354 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:20.580763 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:20.580776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.583220 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.583228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:20.586072 (+ 2844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:20.586091 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.589714 (+ 3623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.589723 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:20.590042 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.590112 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 06:54:20.591576 (+ 1464us) client_negotiation.cc:240] Negotiation successful
0927 06:54:20.591820 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1804,"thread_start_us":1718,"threads_started":1}
I20260927 06:54:20.597640 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 5bba34f23cd64d138abcfb37a204b74a (DEFAULT_TABLE table=test-table [id=d472d501ca6f4187b039ae8433f82fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:20.599023 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bba34f23cd64d138abcfb37a204b74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:20.600353 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.580334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35967 (local address 127.0.0.1:49508)
0927 06:54:20.592526 (+ 12192us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:20.592549 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:20.592682 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:20.593847 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:20.593855 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:20.593889 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:20.594341 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:20.594355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.596099 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.596105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:20.597646 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:20.597657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.599641 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.599647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:20.599817 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.599846 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 06:54:20.599951 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 06:54:20.600056 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12024,"thread_start_us":13683,"threads_started":1}
I20260927 06:54:20.600389 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.580418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49508 (local address 127.0.97.3:35967)
0927 06:54:20.593288 (+ 12870us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:20.593294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:20.593319 (+ 25us) server_negotiation.cc:407] Connection header received
0927 06:54:20.593384 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:20.593390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:20.593674 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:20.593819 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:20.594550 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.595956 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.597789 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.599467 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.599613 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.599830 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:20.599986 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 06:54:20.600115 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12724,"thread_start_us":12542,"threads_started":1}
I20260927 06:54:20.601195 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.579523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35601 (local address 127.0.0.1:39880)
0927 06:54:20.579973 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:20.579992 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:20.580091 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:20.594985 (+ 14894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:20.594989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:20.595013 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:20.595375 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:20.595384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.596976 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.596981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:20.598993 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:20.599007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.600538 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.600544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:20.600714 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.600742 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 06:54:20.600798 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 06:54:20.600912 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":299,"threads_started":1}
I20260927 06:54:20.602280 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.579997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39880 (local address 127.0.97.2:35601)
0927 06:54:20.594436 (+ 14439us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:20.594446 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:20.594474 (+ 28us) server_negotiation.cc:407] Connection header received
0927 06:54:20.594561 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:20.594567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:20.594785 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:20.595511 (+ 726us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:20.595653 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.596833 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.599180 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.600352 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.601578 (+ 1226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.601763 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:20.601823 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 06:54:20.601970 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14281,"thread_start_us":14147,"threads_started":1}
I20260927 06:54:20.603307 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 5bba34f23cd64d138abcfb37a204b74a (DEFAULT_TABLE table=test-table [id=d472d501ca6f4187b039ae8433f82fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:20.604645 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bba34f23cd64d138abcfb37a204b74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:20.605377 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 5bba34f23cd64d138abcfb37a204b74a (DEFAULT_TABLE table=test-table [id=d472d501ca6f4187b039ae8433f82fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:20.606616 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bba34f23cd64d138abcfb37a204b74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:20.610553 2014 tablet_bootstrap.cc:492] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4: Bootstrap starting.
I20260927 06:54:20.613571 2014 tablet_bootstrap.cc:654] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:20.615279 2014 log.cc:824] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:20.617765 2014 tablet_bootstrap.cc:492] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4: No bootstrap required, opened a new log
I20260927 06:54:20.618131 2014 ts_tablet_manager.cc:1397] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 06:54:20.618165 2016 tablet_bootstrap.cc:492] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0: Bootstrap starting.
I20260927 06:54:20.620123 2017 tablet_bootstrap.cc:492] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a: Bootstrap starting.
I20260927 06:54:20.621603 2016 tablet_bootstrap.cc:654] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:20.622848 2016 log.cc:824] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:20.623410 2017 tablet_bootstrap.cc:654] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:20.624647 2017 log.cc:824] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:20.626245 2016 tablet_bootstrap.cc:492] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0: No bootstrap required, opened a new log
I20260927 06:54:20.626492 2016 ts_tablet_manager.cc:1397] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 06:54:20.627187 2017 tablet_bootstrap.cc:492] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a: No bootstrap required, opened a new log
I20260927 06:54:20.627434 2017 ts_tablet_manager.cc:1397] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260927 06:54:20.630115 2014 raft_consensus.cc:348] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:20.630509 2014 raft_consensus.cc:374] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:20.630643 2014 raft_consensus.cc:729] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835f92c8c5d54d1f9d937d1812a78ee4, State: Initialized, Role: FOLLOWER
I20260927 06:54:20.631233 2014 consensus_queue.cc:260] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:20.632915 2014 ts_tablet_manager.cc:1428] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260927 06:54:20.635052 2020 raft_consensus.cc:482] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:54:20.635345 2020 raft_consensus.cc:504] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:20.636302 2017 raft_consensus.cc:348] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:20.636729 2017 raft_consensus.cc:374] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:20.636861 2017 raft_consensus.cc:729] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f98fcd5b83b4e6ab4e29353bc59204a, State: Initialized, Role: FOLLOWER
I20260927 06:54:20.636960 2016 raft_consensus.cc:348] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:20.637373 2016 raft_consensus.cc:374] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:20.637503 2016 raft_consensus.cc:729] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e77fecbe105443c94a94a44c2b1f1d0, State: Initialized, Role: FOLLOWER
I20260927 06:54:20.637754 2017 consensus_queue.cc:260] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:20.638207 2016 consensus_queue.cc:260] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:20.639580 1995 heartbeater.cc:503] Master 127.0.97.62:42255 was elected leader, sending a full tablet report...
I20260927 06:54:20.640379 2016 ts_tablet_manager.cc:1428] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260927 06:54:20.643522 2020 leader_election.cc:288] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f98fcd5b83b4e6ab4e29353bc59204a (127.0.97.2:35601), 0e77fecbe105443c94a94a44c2b1f1d0 (127.0.97.3:35967)
I20260927 06:54:20.645947 1856 heartbeater.cc:503] Master 127.0.97.62:42255 was elected leader, sending a full tablet report...
I20260927 06:54:20.646560 2017 ts_tablet_manager.cc:1428] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a: Time spent starting tablet: real 0.019s user 0.015s sys 0.003s
I20260927 06:54:20.655229 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.637507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42879 (local address 127.0.97.2:35601)
0927 06:54:20.637854 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:20.637860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:20.647832 (+ 9972us) server_negotiation.cc:407] Connection header received
0927 06:54:20.648115 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:20.648124 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:20.648218 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:20.648292 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:20.649209 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.650315 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.652992 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.654207 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.654335 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.654434 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:20.654842 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 06:54:20.654985 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260927 06:54:20.655303 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.637211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35601 (local address 127.0.97.1:42879)
0927 06:54:20.647742 (+ 10531us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:20.647764 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:20.647934 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:20.648533 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:20.648541 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:20.648582 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:20.649053 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:20.649068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.650444 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.650449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:20.652850 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:20.652861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.654568 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.654578 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:20.654691 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.654712 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 06:54:20.654820 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 06:54:20.654941 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10399,"mutex_wait_us":21,"thread_start_us":10259,"threads_started":1}
I20260927 06:54:20.656208 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.647962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41977 (local address 127.0.97.3:35967)
0927 06:54:20.648129 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:20.648137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:20.649750 (+ 1613us) server_negotiation.cc:407] Connection header received
0927 06:54:20.649974 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:20.649978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:20.650090 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:20.650195 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:20.650860 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.651931 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.654520 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.655250 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.655395 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.655497 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:20.655953 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 06:54:20.656043 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 06:54:20.656409 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.647871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35967 (local address 127.0.97.1:41977)
0927 06:54:20.649664 (+ 1793us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:20.649681 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:20.649837 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:20.650228 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:20.650233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:20.650257 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:20.650713 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:20.650722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.652045 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.652048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:20.654361 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:20.654374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.655644 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.655648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:20.655781 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.655816 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 06:54:20.655909 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 06:54:20.656047 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1665,"mutex_wait_us":3,"thread_start_us":1510,"threads_started":1}
I20260927 06:54:20.656236 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bba34f23cd64d138abcfb37a204b74a" candidate_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" is_pre_election: true
I20260927 06:54:20.656719 1811 raft_consensus.cc:2454] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835f92c8c5d54d1f9d937d1812a78ee4 in term 0.
I20260927 06:54:20.657284 1600 leader_election.cc:302] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f98fcd5b83b4e6ab4e29353bc59204a, 835f92c8c5d54d1f9d937d1812a78ee4; no voters:
I20260927 06:54:20.657351 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bba34f23cd64d138abcfb37a204b74a" candidate_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" is_pre_election: true
I20260927 06:54:20.657658 2020 raft_consensus.cc:2781] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:54:20.657791 2020 raft_consensus.cc:482] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:54:20.657908 2020 raft_consensus.cc:3037] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:20.657827 1950 raft_consensus.cc:2454] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835f92c8c5d54d1f9d937d1812a78ee4 in term 0.
I20260927 06:54:20.660205 2020 raft_consensus.cc:504] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:20.660797 2020 leader_election.cc:288] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [CANDIDATE]: Term 1 election: Requested vote from peers 3f98fcd5b83b4e6ab4e29353bc59204a (127.0.97.2:35601), 0e77fecbe105443c94a94a44c2b1f1d0 (127.0.97.3:35967)
I20260927 06:54:20.661186 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bba34f23cd64d138abcfb37a204b74a" candidate_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a"
I20260927 06:54:20.661243 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bba34f23cd64d138abcfb37a204b74a" candidate_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e77fecbe105443c94a94a44c2b1f1d0"
I20260927 06:54:20.661392 1811 raft_consensus.cc:3037] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:20.661427 1950 raft_consensus.cc:3037] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:20.663637 1950 raft_consensus.cc:2454] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835f92c8c5d54d1f9d937d1812a78ee4 in term 1.
I20260927 06:54:20.663990 1600 leader_election.cc:302] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e77fecbe105443c94a94a44c2b1f1d0, 835f92c8c5d54d1f9d937d1812a78ee4; no voters:
I20260927 06:54:20.664218 1811 raft_consensus.cc:2454] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835f92c8c5d54d1f9d937d1812a78ee4 in term 1.
I20260927 06:54:20.664247 2020 raft_consensus.cc:2781] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:20.664742 2020 raft_consensus.cc:686] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 1 LEADER]: Becoming Leader. State: Replica: 835f92c8c5d54d1f9d937d1812a78ee4, State: Running, Role: LEADER
I20260927 06:54:20.665222 2020 consensus_queue.cc:237] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:20.669688 1510 catalog_manager.cc:5495] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 835f92c8c5d54d1f9d937d1812a78ee4 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } health_report { overall_health: UNKNOWN } } }
I20260927 06:54:20.689249 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.684142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50270 (local address 127.0.97.1:38011)
0927 06:54:20.684320 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:20.684327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:20.684417 (+ 90us) server_negotiation.cc:407] Connection header received
0927 06:54:20.684722 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:20.684734 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:20.684819 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:20.684950 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:54:20.685311 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.685975 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.687260 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.687632 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.687757 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.688203 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:54:20.688652 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:54:20.688808 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:20.688936 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 06:54:20.689006 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 06:54:20.696246 1950 raft_consensus.cc:1261] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0 [term 1 FOLLOWER]: Refusing update from remote peer 835f92c8c5d54d1f9d937d1812a78ee4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:54:20.696310 1811 raft_consensus.cc:1261] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 1 FOLLOWER]: Refusing update from remote peer 835f92c8c5d54d1f9d937d1812a78ee4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:54:20.697193 2025 consensus_queue.cc:1035] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:20.697604 2020 consensus_queue.cc:1035] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:20.708220 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333855480603590656 to 7333855480488419328. Current Snapshot: MvccSnapshot[applied={T|T < 7333855480603590656}]
I20260927 06:54:20.726143 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:54:20.726445 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:54:20.730253 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6: Sending DeleteTablet for 3 replicas of tablet 5bba34f23cd64d138abcfb37a204b74a
I20260927 06:54:20.731514 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bba34f23cd64d138abcfb37a204b74a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50268
I20260927 06:54:20.731493 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bba34f23cd64d138abcfb37a204b74a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49508
I20260927 06:54:20.731765 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bba34f23cd64d138abcfb37a204b74a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39880
I20260927 06:54:20.734799 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:54:20.735404 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:54:20.743139 2040 tablet_replica.cc:330] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a: stopping tablet replica
I20260927 06:54:20.745062 1924 tablet_service.cc:1467] Processing CreateTablet for tablet e91e7de6eddf48ca91a9dbb571314bed (DEFAULT_TABLE table=test-table [id=507cd718d5a64092b0fe6a69e597ae83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:20.745182 1646 tablet_service.cc:1467] Processing CreateTablet for tablet e91e7de6eddf48ca91a9dbb571314bed (DEFAULT_TABLE table=test-table [id=507cd718d5a64092b0fe6a69e597ae83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:20.745635 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e91e7de6eddf48ca91a9dbb571314bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:20.745841 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e91e7de6eddf48ca91a9dbb571314bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:20.745007 1786 tablet_service.cc:1467] Processing CreateTablet for tablet e91e7de6eddf48ca91a9dbb571314bed (DEFAULT_TABLE table=test-table [id=507cd718d5a64092b0fe6a69e597ae83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:20.750679 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e91e7de6eddf48ca91a9dbb571314bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:20.751643 2016 tablet_bootstrap.cc:492] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0: Bootstrap starting.
I20260927 06:54:20.753278 2016 tablet_bootstrap.cc:654] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:20.754714 2040 raft_consensus.cc:2229] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:54:20.755255 2040 raft_consensus.cc:2258] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:20.755879 2038 tablet_replica.cc:330] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0: stopping tablet replica
I20260927 06:54:20.756316 2038 raft_consensus.cc:2229] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:54:20.756764 2038 raft_consensus.cc:2258] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:20.757802 2014 tablet_bootstrap.cc:492] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4: Bootstrap starting.
I20260927 06:54:20.758649 2040 ts_tablet_manager.cc:1905] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:20.758908 2039 tablet_replica.cc:330] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4: stopping tablet replica
I20260927 06:54:20.759454 2014 tablet_bootstrap.cc:654] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:20.759414 2039 raft_consensus.cc:2229] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:54:20.760186 2039 raft_consensus.cc:2258] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:20.762040 2017 tablet_bootstrap.cc:492] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a: Bootstrap starting.
I20260927 06:54:20.764108 2017 tablet_bootstrap.cc:654] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:20.766081 2016 tablet_bootstrap.cc:492] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0: No bootstrap required, opened a new log
I20260927 06:54:20.766338 2016 ts_tablet_manager.cc:1397] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20260927 06:54:20.766383 2039 ts_tablet_manager.cc:1905] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:20.766489 2040 ts_tablet_manager.cc:1918] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:54:20.766724 2040 log.cc:1196] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/5bba34f23cd64d138abcfb37a204b74a
I20260927 06:54:20.767202 2016 raft_consensus.cc:348] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:20.767382 2040 ts_tablet_manager.cc:1939] T 5bba34f23cd64d138abcfb37a204b74a P 3f98fcd5b83b4e6ab4e29353bc59204a: Deleting consensus metadata
I20260927 06:54:20.767530 2016 raft_consensus.cc:374] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:20.767668 2016 raft_consensus.cc:729] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e77fecbe105443c94a94a44c2b1f1d0, State: Initialized, Role: FOLLOWER
I20260927 06:54:20.768011 2016 consensus_queue.cc:260] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:20.769405 1497 catalog_manager.cc:4841] TS 3f98fcd5b83b4e6ab4e29353bc59204a (127.0.97.2:35601): tablet 5bba34f23cd64d138abcfb37a204b74a (table test-table [id=d472d501ca6f4187b039ae8433f82fbb]) successfully deleted
I20260927 06:54:20.770201 2016 ts_tablet_manager.cc:1428] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 06:54:20.771772 2017 tablet_bootstrap.cc:492] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a: No bootstrap required, opened a new log
I20260927 06:54:20.771962 2017 ts_tablet_manager.cc:1397] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260927 06:54:20.772794 2017 raft_consensus.cc:348] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:20.773061 2017 raft_consensus.cc:374] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:20.773192 2017 raft_consensus.cc:729] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f98fcd5b83b4e6ab4e29353bc59204a, State: Initialized, Role: FOLLOWER
I20260927 06:54:20.773514 2017 consensus_queue.cc:260] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:20.774060 2039 ts_tablet_manager.cc:1918] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:54:20.774238 2039 log.cc:1196] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/5bba34f23cd64d138abcfb37a204b74a
I20260927 06:54:20.774783 2039 ts_tablet_manager.cc:1939] T 5bba34f23cd64d138abcfb37a204b74a P 835f92c8c5d54d1f9d937d1812a78ee4: Deleting consensus metadata
I20260927 06:54:20.774801 2017 ts_tablet_manager.cc:1428] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 06:54:20.776450 1497 catalog_manager.cc:4841] TS 835f92c8c5d54d1f9d937d1812a78ee4 (127.0.97.1:38011): tablet 5bba34f23cd64d138abcfb37a204b74a (table test-table [id=d472d501ca6f4187b039ae8433f82fbb]) successfully deleted
I20260927 06:54:20.777910 2038 ts_tablet_manager.cc:1905] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:20.779211 2014 tablet_bootstrap.cc:492] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4: No bootstrap required, opened a new log
I20260927 06:54:20.779366 2014 ts_tablet_manager.cc:1397] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.000s
I20260927 06:54:20.780185 2014 raft_consensus.cc:348] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:20.780445 2014 raft_consensus.cc:374] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:20.780578 2014 raft_consensus.cc:729] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835f92c8c5d54d1f9d937d1812a78ee4, State: Initialized, Role: FOLLOWER
I20260927 06:54:20.780882 2014 consensus_queue.cc:260] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:20.781829 2014 ts_tablet_manager.cc:1428] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 06:54:20.785303 2038 ts_tablet_manager.cc:1918] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:54:20.785457 2038 log.cc:1196] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/5bba34f23cd64d138abcfb37a204b74a
I20260927 06:54:20.785898 2038 ts_tablet_manager.cc:1939] T 5bba34f23cd64d138abcfb37a204b74a P 0e77fecbe105443c94a94a44c2b1f1d0: Deleting consensus metadata
I20260927 06:54:20.787500 1497 catalog_manager.cc:4841] TS 0e77fecbe105443c94a94a44c2b1f1d0 (127.0.97.3:35967): tablet 5bba34f23cd64d138abcfb37a204b74a (table test-table [id=d472d501ca6f4187b039ae8433f82fbb]) successfully deleted
I20260927 06:54:20.955451 2023 raft_consensus.cc:482] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:54:20.955669 2023 raft_consensus.cc:504] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:20.957635 2023 leader_election.cc:288] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835f92c8c5d54d1f9d937d1812a78ee4 (127.0.97.1:38011), 3f98fcd5b83b4e6ab4e29353bc59204a (127.0.97.2:35601)
I20260927 06:54:20.964316 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.957789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38011 (local address 127.0.97.3:42311)
0927 06:54:20.958107 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:20.958127 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:20.958232 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:20.958894 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:20.958898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:20.958920 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:20.959112 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:20.959120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.960384 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.960390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:20.962466 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:20.962480 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.963582 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.963587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:20.963762 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.963784 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 06:54:20.963879 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 06:54:20.964031 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260927 06:54:20.964775 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.957925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42311 (local address 127.0.97.1:38011)
0927 06:54:20.958166 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:20.958173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:20.958199 (+ 26us) server_negotiation.cc:407] Connection header received
0927 06:54:20.958442 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:20.958460 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:20.958578 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:20.958690 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:20.959252 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.960253 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.962646 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.963399 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.963577 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.963677 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:20.963921 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 06:54:20.964053 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 06:54:20.965402 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.958485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56323 (local address 127.0.97.2:35601)
0927 06:54:20.958660 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:20.958666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:20.959212 (+ 546us) server_negotiation.cc:407] Connection header received
0927 06:54:20.959461 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:20.959475 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:20.959575 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:20.959664 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:20.960199 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.960987 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.962715 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.963870 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.963995 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.964104 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:20.964990 (+ 886us) server_negotiation.cc:299] Negotiation successful
0927 06:54:20.965127 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 06:54:20.965797 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91e7de6eddf48ca91a9dbb571314bed" candidate_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" is_pre_election: true
I20260927 06:54:20.966130 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:20.958164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35601 (local address 127.0.97.3:56323)
0927 06:54:20.959107 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:20.959126 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:20.959311 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:20.959707 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:20.959713 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:20.959743 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:20.960030 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:20.960040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.961123 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.961130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:20.962496 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:20.962507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:20.964249 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:20.964256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:20.964808 (+ 552us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:20.964842 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 06:54:20.965626 (+ 784us) client_negotiation.cc:240] Negotiation successful
0927 06:54:20.965763 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":622,"threads_started":1}
I20260927 06:54:20.966259 1671 raft_consensus.cc:2454] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e77fecbe105443c94a94a44c2b1f1d0 in term 0.
I20260927 06:54:20.966774 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91e7de6eddf48ca91a9dbb571314bed" candidate_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" is_pre_election: true
I20260927 06:54:20.966998 1811 raft_consensus.cc:2454] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e77fecbe105443c94a94a44c2b1f1d0 in term 0.
I20260927 06:54:20.966910 1879 leader_election.cc:302] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e77fecbe105443c94a94a44c2b1f1d0, 835f92c8c5d54d1f9d937d1812a78ee4; no voters:
I20260927 06:54:20.967365 2023 raft_consensus.cc:2781] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:54:20.967499 2023 raft_consensus.cc:482] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:54:20.967608 2023 raft_consensus.cc:3037] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:20.969662 2023 raft_consensus.cc:504] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:20.970263 2023 leader_election.cc:288] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [CANDIDATE]: Term 1 election: Requested vote from peers 835f92c8c5d54d1f9d937d1812a78ee4 (127.0.97.1:38011), 3f98fcd5b83b4e6ab4e29353bc59204a (127.0.97.2:35601)
I20260927 06:54:20.970623 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91e7de6eddf48ca91a9dbb571314bed" candidate_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835f92c8c5d54d1f9d937d1812a78ee4"
I20260927 06:54:20.970788 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91e7de6eddf48ca91a9dbb571314bed" candidate_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a"
I20260927 06:54:20.970886 1671 raft_consensus.cc:3037] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:20.970985 1811 raft_consensus.cc:3037] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:20.973747 1671 raft_consensus.cc:2454] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e77fecbe105443c94a94a44c2b1f1d0 in term 1.
I20260927 06:54:20.973742 1811 raft_consensus.cc:2454] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e77fecbe105443c94a94a44c2b1f1d0 in term 1.
I20260927 06:54:20.974143 1879 leader_election.cc:302] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e77fecbe105443c94a94a44c2b1f1d0, 3f98fcd5b83b4e6ab4e29353bc59204a; no voters:
I20260927 06:54:20.974426 2023 raft_consensus.cc:2781] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:20.974888 2023 raft_consensus.cc:686] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [term 1 LEADER]: Becoming Leader. State: Replica: 0e77fecbe105443c94a94a44c2b1f1d0, State: Running, Role: LEADER
I20260927 06:54:20.975277 2023 consensus_queue.cc:237] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:20.979070 1510 catalog_manager.cc:5495] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e77fecbe105443c94a94a44c2b1f1d0 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } health_report { overall_health: HEALTHY } } }
W20260927 06:54:21.005599 1996 tablet.cc:2369] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:54:21.013077 1857 tablet.cc:2369] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:54:21.016623 1718 tablet.cc:2369] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 06:54:21 dist-test-slave-f892 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:21.037813 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:21.020937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42255 (local address 127.0.0.1:48078)
0927 06:54:21.021617 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:21.021637 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:21.021809 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:21.022596 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:21.022600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:21.023410 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:21.023761 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:21.023773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.025273 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.025278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:21.026837 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:21.026860 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.028020 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.028026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:21.028196 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:21.029030 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:21.029057 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:21.032232 (+ 3175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:21.035381 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:21.035392 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:21.035400 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:21.035765 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:21.036116 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:21.036120 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:21.036124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:21.036262 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:21.036759 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:21.036767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:21.036924 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 06:54:21.037101 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 06:54:21.037376 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":381,"threads_started":1}
I20260927 06:54:21.037842 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:21.020852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.0.97.62:42255)
0927 06:54:21.021032 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:21.021038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:21.021766 (+ 728us) server_negotiation.cc:407] Connection header received
0927 06:54:21.021939 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:21.021944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:21.022435 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:21.022555 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:21.023918 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.025132 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.026996 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.027651 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.027807 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:21.032405 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:21.032415 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:21.032421 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:21.032484 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:21.035246 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:21.035879 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:21.035886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:21.035891 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:21.036006 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:21.036362 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:21.036369 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:21.036373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:21.036632 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:21.036773 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:21.037175 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 06:54:21.037546 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":51}
I20260927 06:54:21.043407 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48078:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 06:54:21.044390 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:54:21.051852 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 21fc537e7abe4035ba8cd8a79ac4051a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=05867309e22c49f6a24c88f583dbf6f6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 06:54:21.052021 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 21fc537e7abe4035ba8cd8a79ac4051a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=05867309e22c49f6a24c88f583dbf6f6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 06:54:21.052495 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21fc537e7abe4035ba8cd8a79ac4051a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:21.052605 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21fc537e7abe4035ba8cd8a79ac4051a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:21.059329 2017 tablet_bootstrap.cc:492] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a: Bootstrap starting.
I20260927 06:54:21.059726 2016 tablet_bootstrap.cc:492] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0: Bootstrap starting.
I20260927 06:54:21.060220 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 21fc537e7abe4035ba8cd8a79ac4051a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=05867309e22c49f6a24c88f583dbf6f6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 06:54:21.060786 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21fc537e7abe4035ba8cd8a79ac4051a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:21.061182 2017 tablet_bootstrap.cc:654] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:21.061357 2016 tablet_bootstrap.cc:654] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:21.071408 2014 tablet_bootstrap.cc:492] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4: Bootstrap starting.
I20260927 06:54:21.073227 2014 tablet_bootstrap.cc:654] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:21.073567 2017 tablet_bootstrap.cc:492] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a: No bootstrap required, opened a new log
I20260927 06:54:21.073716 2017 ts_tablet_manager.cc:1397] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260927 06:54:21.074231 2016 tablet_bootstrap.cc:492] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0: No bootstrap required, opened a new log
I20260927 06:54:21.074383 2016 ts_tablet_manager.cc:1397] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.003s
I20260927 06:54:21.074476 2017 raft_consensus.cc:348] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:21.074793 2017 raft_consensus.cc:374] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:21.074923 2017 raft_consensus.cc:729] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f98fcd5b83b4e6ab4e29353bc59204a, State: Initialized, Role: FOLLOWER
I20260927 06:54:21.075311 2017 consensus_queue.cc:260] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:21.075930 2017 ts_tablet_manager.cc:1428] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 06:54:21.075848 2022 tablet_replica.cc:439] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } } }
I20260927 06:54:21.076071 2016 raft_consensus.cc:348] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:21.076208 2022 tablet_replica.cc:442] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:54:21.076274 2016 raft_consensus.cc:374] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:21.076323 2016 raft_consensus.cc:729] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e77fecbe105443c94a94a44c2b1f1d0, State: Initialized, Role: FOLLOWER
I20260927 06:54:21.076532 2016 consensus_queue.cc:260] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:21.077000 2016 ts_tablet_manager.cc:1428] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260927 06:54:21.076884 2023 tablet_replica.cc:439] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } } }
I20260927 06:54:21.077183 2023 tablet_replica.cc:442] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:54:21.081073 2014 tablet_bootstrap.cc:492] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4: No bootstrap required, opened a new log
I20260927 06:54:21.081260 2014 ts_tablet_manager.cc:1397] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 06:54:21.081843 2014 raft_consensus.cc:348] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:21.082068 2014 raft_consensus.cc:374] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:21.082170 2014 raft_consensus.cc:729] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835f92c8c5d54d1f9d937d1812a78ee4, State: Initialized, Role: FOLLOWER
I20260927 06:54:21.082408 2014 consensus_queue.cc:260] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:21.082795 2025 tablet_replica.cc:439] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } } }
I20260927 06:54:21.083156 2025 tablet_replica.cc:442] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:54:21.082994 2014 ts_tablet_manager.cc:1428] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 06:54:21.264998 2022 raft_consensus.cc:482] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:54:21.265220 2022 raft_consensus.cc:504] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:21.266718 2022 leader_election.cc:288] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835f92c8c5d54d1f9d937d1812a78ee4 (127.0.97.1:38011), 0e77fecbe105443c94a94a44c2b1f1d0 (127.0.97.3:35967)
I20260927 06:54:21.273263 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:21.267083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38011 (local address 127.0.97.2:51339)
0927 06:54:21.267529 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:21.267544 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:21.267699 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:21.268166 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:21.268174 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:21.268216 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:21.268734 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:21.268745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.269734 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.269741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:21.271504 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:21.271516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.272702 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.272706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:21.272818 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:21.272842 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 06:54:21.272919 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 06:54:21.273014 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":226,"threads_started":1}
I20260927 06:54:21.273347 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:21.267215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51339 (local address 127.0.97.1:38011)
0927 06:54:21.267435 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:21.267441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:21.267629 (+ 188us) server_negotiation.cc:407] Connection header received
0927 06:54:21.267863 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:21.267872 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:21.267980 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:21.268132 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:21.268863 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.269606 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.271646 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.272541 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.272671 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:21.272764 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:21.272966 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 06:54:21.273101 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 06:54:21.274206 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fc537e7abe4035ba8cd8a79ac4051a" candidate_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" is_pre_election: true
I20260927 06:54:21.274331 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:21.267891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38445 (local address 127.0.97.3:35967)
0927 06:54:21.268900 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:21.268908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:21.268936 (+ 28us) server_negotiation.cc:407] Connection header received
0927 06:54:21.269008 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:21.269012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:21.269108 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:21.269217 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:21.269686 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.270525 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.272299 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.273434 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.273570 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:21.273671 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:21.273966 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 06:54:21.274145 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":884,"thread_start_us":791,"threads_started":1}
I20260927 06:54:21.274384 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:21.267793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35967 (local address 127.0.97.2:38445)
0927 06:54:21.268729 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:21.268748 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:21.268850 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:21.269240 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:21.269244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:21.269268 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:21.269539 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:21.269547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.270650 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.270657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:21.272121 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:21.272133 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.273638 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.273644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:21.273816 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:21.273837 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 06:54:21.273905 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 06:54:21.274075 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":633,"threads_started":1}
I20260927 06:54:21.274471 1671 raft_consensus.cc:2454] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f98fcd5b83b4e6ab4e29353bc59204a in term 0.
I20260927 06:54:21.274915 1740 leader_election.cc:302] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f98fcd5b83b4e6ab4e29353bc59204a, 835f92c8c5d54d1f9d937d1812a78ee4; no voters:
I20260927 06:54:21.275106 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fc537e7abe4035ba8cd8a79ac4051a" candidate_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" is_pre_election: true
I20260927 06:54:21.275310 2022 raft_consensus.cc:2781] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:54:21.275352 1950 raft_consensus.cc:2454] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f98fcd5b83b4e6ab4e29353bc59204a in term 0.
I20260927 06:54:21.275437 2022 raft_consensus.cc:482] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:54:21.275532 2022 raft_consensus.cc:3037] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:21.277568 2022 raft_consensus.cc:504] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:21.278167 2022 leader_election.cc:288] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [CANDIDATE]: Term 1 election: Requested vote from peers 835f92c8c5d54d1f9d937d1812a78ee4 (127.0.97.1:38011), 0e77fecbe105443c94a94a44c2b1f1d0 (127.0.97.3:35967)
I20260927 06:54:21.278580 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fc537e7abe4035ba8cd8a79ac4051a" candidate_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835f92c8c5d54d1f9d937d1812a78ee4"
I20260927 06:54:21.278743 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fc537e7abe4035ba8cd8a79ac4051a" candidate_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e77fecbe105443c94a94a44c2b1f1d0"
I20260927 06:54:21.278833 1671 raft_consensus.cc:3037] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:21.278950 1950 raft_consensus.cc:3037] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:21.280727 1671 raft_consensus.cc:2454] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f98fcd5b83b4e6ab4e29353bc59204a in term 1.
I20260927 06:54:21.280872 1950 raft_consensus.cc:2454] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f98fcd5b83b4e6ab4e29353bc59204a in term 1.
I20260927 06:54:21.281086 1740 leader_election.cc:302] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f98fcd5b83b4e6ab4e29353bc59204a, 835f92c8c5d54d1f9d937d1812a78ee4; no voters:
I20260927 06:54:21.281373 2022 raft_consensus.cc:2781] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:21.281739 2022 raft_consensus.cc:686] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [term 1 LEADER]: Becoming Leader. State: Replica: 3f98fcd5b83b4e6ab4e29353bc59204a, State: Running, Role: LEADER
I20260927 06:54:21.282172 2022 consensus_queue.cc:237] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } }
I20260927 06:54:21.283957 2059 tablet_replica.cc:439] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a: TxnStatusTablet state changed. Reason: New leader 3f98fcd5b83b4e6ab4e29353bc59204a. Latest consensus state: current_term: 1 leader_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } } }
I20260927 06:54:21.284348 2059 tablet_replica.cc:442] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a: This TxnStatusTablet replica's current role is: LEADER
I20260927 06:54:21.285578 2061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 06:54:21.285776 2061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 06:54:21.286159 1510 catalog_manager.cc:5495] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f98fcd5b83b4e6ab4e29353bc59204a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } health_report { overall_health: UNKNOWN } } }
I20260927 06:54:21.305064 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:21.299788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.0.97.2:35601)
0927 06:54:21.299941 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:21.299951 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:21.299973 (+ 22us) server_negotiation.cc:407] Connection header received
0927 06:54:21.300275 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:21.300279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:21.300355 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:21.300490 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:54:21.300862 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.301649 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.302968 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.303382 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.303524 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:21.303909 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:54:21.304449 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:54:21.304608 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:21.304682 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 06:54:21.304753 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 06:54:21.305094 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:21.299694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35601 (local address 127.0.0.1:39892)
0927 06:54:21.299907 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:21.299927 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:21.300068 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:21.300521 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:21.300526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:21.300537 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 06:54:21.300732 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:21.300740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.301787 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.301795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:21.302830 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:21.302840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.303535 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.303540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:21.303678 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:21.303727 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 06:54:21.304565 (+ 838us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 06:54:21.304584 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 06:54:21.304667 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 06:54:21.304806 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 06:54:21.309772 1950 raft_consensus.cc:1261] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0 [term 1 FOLLOWER]: Refusing update from remote peer 3f98fcd5b83b4e6ab4e29353bc59204a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:54:21.309830 1671 raft_consensus.cc:1261] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4 [term 1 FOLLOWER]: Refusing update from remote peer 3f98fcd5b83b4e6ab4e29353bc59204a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:54:21.310510 2022 consensus_queue.cc:1035] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:21.310824 2059 consensus_queue.cc:1035] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a [LEADER]: Connected to new peer: Peer: permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:21.319804 2023 tablet_replica.cc:439] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0: TxnStatusTablet state changed. Reason: New leader 3f98fcd5b83b4e6ab4e29353bc59204a. Latest consensus state: current_term: 1 leader_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } } }
I20260927 06:54:21.320097 2023 tablet_replica.cc:442] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:54:21.322696 2025 tablet_replica.cc:439] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4: TxnStatusTablet state changed. Reason: New leader 3f98fcd5b83b4e6ab4e29353bc59204a. Latest consensus state: current_term: 1 leader_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } } }
I20260927 06:54:21.322959 2025 tablet_replica.cc:442] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:54:21.323323 2064 mvcc.cc:205] Tried to move back new op lower bound from 7333855483117629440 to 7333855483015266304. Current Snapshot: MvccSnapshot[applied={T|T < 7333855483117629440}]
I20260927 06:54:21.324265 2066 mvcc.cc:205] Tried to move back new op lower bound from 7333855483117629440 to 7333855483015266304. Current Snapshot: MvccSnapshot[applied={T|T < 7333855483117629440}]
I20260927 06:54:21.333848 2023 tablet_replica.cc:439] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } } }
I20260927 06:54:21.334175 2023 tablet_replica.cc:442] T 21fc537e7abe4035ba8cd8a79ac4051a P 0e77fecbe105443c94a94a44c2b1f1d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:54:21.334467 2022 tablet_replica.cc:439] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } } }
I20260927 06:54:21.334982 2025 tablet_replica.cc:439] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } } }
I20260927 06:54:21.335258 2025 tablet_replica.cc:442] T 21fc537e7abe4035ba8cd8a79ac4051a P 835f92c8c5d54d1f9d937d1812a78ee4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:54:21.335290 2022 tablet_replica.cc:442] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a: This TxnStatusTablet replica's current role is: LEADER
I20260927 06:54:21.335618 2022 tablet_replica.cc:439] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 } } peers { permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "0e77fecbe105443c94a94a44c2b1f1d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35967 } } }
I20260927 06:54:21.338193 2022 tablet_replica.cc:442] T 21fc537e7abe4035ba8cd8a79ac4051a P 3f98fcd5b83b4e6ab4e29353bc59204a: This TxnStatusTablet replica's current role is: LEADER
I20260927 06:54:21.352662 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:21.343058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49510 (local address 127.0.97.3:35967)
0927 06:54:21.343329 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:21.343339 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:21.346621 (+ 3282us) server_negotiation.cc:407] Connection header received
0927 06:54:21.346824 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:21.346833 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:21.346952 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:21.347096 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:54:21.347970 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.348647 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.350581 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.350971 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.351137 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:21.351592 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:54:21.352089 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:54:21.352241 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:21.352331 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 06:54:21.352388 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 06:54:21.355870 2023 consensus_queue.cc:1035] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835f92c8c5d54d1f9d937d1812a78ee4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:21.365291 2023 consensus_queue.cc:1035] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f98fcd5b83b4e6ab4e29353bc59204a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:21.383528 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:21.370782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55941 (local address 127.0.97.2:35601)
0927 06:54:21.370966 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:21.370972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:21.378703 (+ 7731us) server_negotiation.cc:407] Connection header received
0927 06:54:21.378891 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:21.378896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:21.378966 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:21.379056 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:54:21.379524 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.380310 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.381650 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.382251 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.382390 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:21.382808 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:54:21.382901 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:54:21.382999 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:21.383149 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 06:54:21.383222 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 06:54:21.383606 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:21.370902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35601 (local address 127.0.97.3:55941)
0927 06:54:21.378577 (+ 7675us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:21.378593 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:21.378745 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:21.379088 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:21.379092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:21.379103 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 06:54:21.379386 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:21.379393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.380437 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.380445 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:21.381502 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:21.381512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.382429 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.382437 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:21.382565 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:21.382618 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 06:54:21.383011 (+ 393us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 06:54:21.383030 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 06:54:21.383131 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 06:54:21.383258 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7501,"thread_start_us":7373,"threads_started":1}
I20260927 06:54:21.428653 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:21.416401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45721 (local address 127.0.97.3:35967)
0927 06:54:21.416583 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:21.416592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:21.422649 (+ 6057us) server_negotiation.cc:407] Connection header received
0927 06:54:21.422860 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:21.422865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:21.422940 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:21.423034 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:54:21.423580 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.425000 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.426653 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.427050 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.427211 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:21.427639 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:54:21.427793 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:54:21.427883 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:21.428216 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 06:54:21.428297 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 06:54:21.429193 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:21.416353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35967 (local address 127.0.97.2:45721)
0927 06:54:21.422542 (+ 6189us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:21.422558 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:21.422726 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:21.423061 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:21.423065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:21.423077 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 06:54:21.423421 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:21.423430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.425121 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.425129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:21.426490 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:21.426521 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.427267 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.427275 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:21.427416 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:21.427468 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 06:54:21.428077 (+ 609us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 06:54:21.428089 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 06:54:21.428186 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 06:54:21.428377 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6011,"thread_start_us":5866,"threads_started":1}
I20260927 06:54:21.470147 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:54:21.470403 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:54:21.474234 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 238c20bc311043d181686d91b29877d6: Sending DeleteTablet for 3 replicas of tablet e91e7de6eddf48ca91a9dbb571314bed
I20260927 06:54:21.475095 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet e91e7de6eddf48ca91a9dbb571314bed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49508
I20260927 06:54:21.475134 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet e91e7de6eddf48ca91a9dbb571314bed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50268
I20260927 06:54:21.475996 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet e91e7de6eddf48ca91a9dbb571314bed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39880
I20260927 06:54:21.487375 2086 tablet_replica.cc:330] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0: stopping tablet replica
I20260927 06:54:21.487696 2086 raft_consensus.cc:2229] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:54:21.488339 2086 raft_consensus.cc:2258] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:21.489363 2086 ts_tablet_manager.cc:1905] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:21.496119 2086 ts_tablet_manager.cc:1918] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:54:21.496309 2086 log.cc:1196] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/e91e7de6eddf48ca91a9dbb571314bed
I20260927 06:54:21.496833 2086 ts_tablet_manager.cc:1939] T e91e7de6eddf48ca91a9dbb571314bed P 0e77fecbe105443c94a94a44c2b1f1d0: Deleting consensus metadata
I20260927 06:54:21.498067 1497 catalog_manager.cc:4841] TS 0e77fecbe105443c94a94a44c2b1f1d0 (127.0.97.3:35967): tablet e91e7de6eddf48ca91a9dbb571314bed (table test-table [id=507cd718d5a64092b0fe6a69e597ae83]) successfully deleted
I20260927 06:54:21.504679 2088 tablet_replica.cc:330] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a: stopping tablet replica
I20260927 06:54:21.505409 2088 raft_consensus.cc:2229] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:54:21.506265 2087 tablet_replica.cc:330] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4: stopping tablet replica
I20260927 06:54:21.506556 2088 raft_consensus.cc:2258] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:21.507782 2088 ts_tablet_manager.cc:1905] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:21.508704 2087 raft_consensus.cc:2229] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:54:21.508996 2087 raft_consensus.cc:2258] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:21.520066 2087 ts_tablet_manager.cc:1905] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:21.524930 2088 ts_tablet_manager.cc:1918] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:54:21.525117 2088 log.cc:1196] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/e91e7de6eddf48ca91a9dbb571314bed
I20260927 06:54:21.525672 2088 ts_tablet_manager.cc:1939] T e91e7de6eddf48ca91a9dbb571314bed P 3f98fcd5b83b4e6ab4e29353bc59204a: Deleting consensus metadata
I20260927 06:54:21.526790 1497 catalog_manager.cc:4841] TS 3f98fcd5b83b4e6ab4e29353bc59204a (127.0.97.2:35601): tablet e91e7de6eddf48ca91a9dbb571314bed (table test-table [id=507cd718d5a64092b0fe6a69e597ae83]) successfully deleted
Sep 27 06:54:21 dist-test-slave-f892 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492060, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:54:21.540469 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:21.490804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50278 (local address 127.0.97.1:38011)
0927 06:54:21.491012 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:21.491021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:21.508506 (+ 17485us) server_negotiation.cc:407] Connection header received
0927 06:54:21.512732 (+ 4226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:21.512738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:21.512822 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:21.512898 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:21.515184 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.516627 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.520031 (+ 3404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.520660 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.520763 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:21.532529 (+ 11766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:21.532543 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:21.532561 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:21.532611 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:21.536865 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:21.537700 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:21.537710 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:21.537714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:21.537860 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:21.538339 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:21.538344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:21.538347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:21.538766 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:21.539155 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:21.539809 (+ 654us) server_negotiation.cc:299] Negotiation successful
0927 06:54:21.540096 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 06:54:21.541447 2087 ts_tablet_manager.cc:1918] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:54:21.541590 2087 log.cc:1196] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/e91e7de6eddf48ca91a9dbb571314bed
I20260927 06:54:21.542529 2087 ts_tablet_manager.cc:1939] T e91e7de6eddf48ca91a9dbb571314bed P 835f92c8c5d54d1f9d937d1812a78ee4: Deleting consensus metadata
W20260927 06:54:21.542482 1712 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50278
I20260927 06:54:21.547557 1497 catalog_manager.cc:4841] TS 835f92c8c5d54d1f9d937d1812a78ee4 (127.0.97.1:38011): tablet e91e7de6eddf48ca91a9dbb571314bed (table test-table [id=507cd718d5a64092b0fe6a69e597ae83]) successfully deleted
I20260927 06:54:21.601995 2092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:21.567250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.0.97.62:42255)
0927 06:54:21.585950 (+ 18700us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:21.585956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:21.585979 (+ 23us) server_negotiation.cc:407] Connection header received
0927 06:54:21.586081 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:21.586087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:21.586169 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:21.586298 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:21.588282 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.589451 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.591191 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.591779 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.591899 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:21.595023 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:21.595030 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:21.595035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:21.595072 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:21.598509 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:21.599646 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:21.599653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:21.599657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:21.599795 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:21.600247 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:21.600252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:21.600255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:21.600617 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:21.600848 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:21.601394 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 06:54:21.601623 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18563,"thread_start_us":18805,"threads_started":1}
W20260927 06:54:21.602622 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48090
I20260927 06:54:21.605870 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 1581
I20260927 06:54:21.623294 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:21.607606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50290 (local address 127.0.97.1:38011)
0927 06:54:21.607767 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:21.607773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:21.607791 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:54:21.607945 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:21.607963 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:21.608047 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:21.608114 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:21.609651 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.610727 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.612455 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.613111 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.613230 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:21.616089 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:21.616095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:21.616101 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:21.616139 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:21.619447 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:21.620517 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:21.620522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:21.620526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:21.620697 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:21.621156 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:21.621162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:21.621165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:21.621477 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:21.621767 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:21.622530 (+ 763us) server_negotiation.cc:299] Negotiation successful
0927 06:54:21.622787 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 06:54:21.623875 1712 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50290
W20260927 06:54:21.624528 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 1581: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:54:21.624714 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 1581
I20260927 06:54:21.654527 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 1721
Sep 27 06:54:21 dist-test-slave-f892 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492060, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:54:21.674885 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:21.655354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.0.97.2:35601)
0927 06:54:21.655549 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:21.655555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:21.655677 (+ 122us) server_negotiation.cc:407] Connection header received
0927 06:54:21.655884 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:21.655890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:21.655967 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:21.656079 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:21.657603 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.658932 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.660846 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.661540 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.661703 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:21.667147 (+ 5444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:21.667167 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:21.667192 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:21.667236 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:21.671506 (+ 4270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:21.672325 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:21.672334 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:21.672337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:21.672528 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:21.672932 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:21.672938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:21.672941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:21.673414 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:21.673757 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:21.674083 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 06:54:21.674303 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 06:54:21.675688 1851 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39908
W20260927 06:54:21.676420 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 1721: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:54:21.676604 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 1721
I20260927 06:54:21.697333 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 1860
Sep 27 06:54:21 dist-test-slave-f892 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492060, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 06:54:21.713829 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:21.698129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49522 (local address 127.0.97.3:35967)
0927 06:54:21.698310 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:21.698317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:21.698348 (+ 31us) server_negotiation.cc:407] Connection header received
0927 06:54:21.698554 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:21.698557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:21.698625 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:21.698705 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:21.700264 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.700970 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.702812 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:21.703244 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:21.703344 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:21.708146 (+ 4802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:21.708157 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:21.708172 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:21.708202 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:21.711148 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:21.711774 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:21.711781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:21.711783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:21.711891 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:21.712265 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:21.712273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:21.712279 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:21.712691 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:21.713019 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:21.713295 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 06:54:21.713529 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 06:54:21.714701 1990 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49522
W20260927 06:54:21.715353 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 1860: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:54:21.715503 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 1860
I20260927 06:54:21.733842 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 1478
I20260927 06:54:21.734969 1550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:21.846578 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 1478
2026-09-27T06:54:21Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4600 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:54:21 dist-test-slave-f892 krb5kdc[2098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:54:21 dist-test-slave-f892 krb5kdc[2098](info): set up 2 sockets
Sep 27 06:54:21 dist-test-slave-f892 krb5kdc[2098](info): commencing operation
krb5kdc: starting...
W20260927 06:54:22.918134 388 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:54:23 dist-test-slave-f892 krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:54:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:54:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:54:23.093526 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38755
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:23.395005 2111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:23.395267 2111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:23.395430 2111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:23.402336 2111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:54:23.402426 2111 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 06:54:23.402516 2111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:23.402634 2111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:23.402714 2111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:54:23.402804 2111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:54:23.411963 2111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38755
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:23.413129 2111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:23.414425 2111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:23.425745 2119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:23.425793 2117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:23.425766 2116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:23.426702 2111 server_base.cc:1034] running on GCE node
I20260927 06:54:23.427506 2111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:23.428869 2111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:23.430136 2111 hybrid_clock.cc:648] HybridClock initialized: now 1790492063430055 us; error 51 us; skew 500 ppm
I20260927 06:54:23.430451 2111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:23 dist-test-slave-f892 krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:23.433508 2111 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:54:23.434890 2111 webserver.cc:460] Webserver started at http://127.0.97.62:45511/ using document root <none> and password file <none>
I20260927 06:54:23.435577 2111 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:23.435702 2111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:23.436008 2111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:23.438894 2111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "c5b85fde79904c3bbbcf6182d8994fa3"
format_stamp: "Formatted at 2026-09-27 06:54:23 on dist-test-slave-f892"
server_key: "823f57d63a3794ec65ba470ebf381ad4"
server_key_iv: "bdefb1f067e8b05cecbc2d7cc673cf37"
server_key_version: "encryptionkey@0"
I20260927 06:54:23.439700 2111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "c5b85fde79904c3bbbcf6182d8994fa3"
format_stamp: "Formatted at 2026-09-27 06:54:23 on dist-test-slave-f892"
server_key: "823f57d63a3794ec65ba470ebf381ad4"
server_key_iv: "bdefb1f067e8b05cecbc2d7cc673cf37"
server_key_version: "encryptionkey@0"
I20260927 06:54:23.444520 2111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 06:54:23.447750 2126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:23.448909 2111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 06:54:23.449096 2111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "c5b85fde79904c3bbbcf6182d8994fa3"
format_stamp: "Formatted at 2026-09-27 06:54:23 on dist-test-slave-f892"
server_key: "823f57d63a3794ec65ba470ebf381ad4"
server_key_iv: "bdefb1f067e8b05cecbc2d7cc673cf37"
server_key_version: "encryptionkey@0"
I20260927 06:54:23.449290 2111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:23.465853 2111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:23.469784 2111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:23.470122 2111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:23.490639 2111 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38755
I20260927 06:54:23.490660 2187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38755 every 8 connection(s)
I20260927 06:54:23.491848 2111 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:54:23.495234 2188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:23.497378 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 2111
I20260927 06:54:23.497582 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:54:23.498258 388 external_mini_cluster.cc:1428] Setting key a8157dfc101dbec64f906d24951230fe
I20260927 06:54:23.506654 2188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3: Bootstrap starting.
I20260927 06:54:23.510279 2188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:23.511510 2188 log.cc:824] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3: Log is configured to *not* fsync() on all Append() calls
Sep 27 06:54:23 dist-test-slave-f892 krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:23.514132 2188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3: No bootstrap required, opened a new log
I20260927 06:54:23.521306 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:23.500117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.0.97.62:38755)
0927 06:54:23.500889 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:23.500899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:23.500937 (+ 38us) server_negotiation.cc:407] Connection header received
0927 06:54:23.502709 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:23.502733 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:23.503097 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:23.503404 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:23.505078 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:23.506355 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:23.508134 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:23.508793 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:23.508911 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:23.513555 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:23.513563 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:23.513576 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:23.513607 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:23.517299 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:23.517893 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:23.517901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:23.517913 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:23.518135 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:23.518464 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:23.518469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:23.518472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:23.519072 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:23.519354 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:23.519662 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 06:54:23.520050 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":358,"threads_started":1}
I20260927 06:54:23.524173 2188 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b85fde79904c3bbbcf6182d8994fa3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38755 } }
I20260927 06:54:23.524474 2188 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:23.524586 2188 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b85fde79904c3bbbcf6182d8994fa3, State: Initialized, Role: FOLLOWER
I20260927 06:54:23.525216 2188 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b85fde79904c3bbbcf6182d8994fa3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38755 } }
I20260927 06:54:23.525513 2188 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:54:23.525664 2188 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:54:23.525820 2188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:23.527747 2188 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b85fde79904c3bbbcf6182d8994fa3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38755 } }
I20260927 06:54:23.528187 2188 leader_election.cc:302] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5b85fde79904c3bbbcf6182d8994fa3; no voters:
I20260927 06:54:23.528743 2188 leader_election.cc:288] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:54:23.528877 2193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:23.529866 2193 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3 [term 1 LEADER]: Becoming Leader. State: Replica: c5b85fde79904c3bbbcf6182d8994fa3, State: Running, Role: LEADER
I20260927 06:54:23.530347 2193 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b85fde79904c3bbbcf6182d8994fa3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38755 } }
I20260927 06:54:23.530640 2188 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:54:23.532689 2195 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5b85fde79904c3bbbcf6182d8994fa3. Latest consensus state: current_term: 1 leader_uuid: "c5b85fde79904c3bbbcf6182d8994fa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b85fde79904c3bbbcf6182d8994fa3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38755 } } }
I20260927 06:54:23.532995 2195 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:23.533599 2194 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5b85fde79904c3bbbcf6182d8994fa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b85fde79904c3bbbcf6182d8994fa3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38755 } } }
I20260927 06:54:23.533852 2194 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:23.535068 2199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:54:23.539851 2199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:54:23.547601 2199 catalog_manager.cc:1347] Generated new cluster ID: cdc7cf62e7f94cd2909f6a08367f79ce
I20260927 06:54:23.547716 2199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:54:23.562866 2199 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:54:23.564112 2199 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:54:23.571228 2199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3: Generated new TSK 0
I20260927 06:54:23.571978 2199 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 06:54:23.617152 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38755
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:23.925130 2214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:23.925418 2214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:23.925619 2214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:23.932698 2214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:23.932844 2214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:23.932998 2214 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 06:54:23.933086 2214 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 06:54:23.933221 2214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:54:23.942934 2214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:23.944239 2214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:23.945510 2214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:23.957885 2222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:23.957909 2219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:23.957990 2220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:23.963305 2214 server_base.cc:1034] running on GCE node
I20260927 06:54:23.963886 2214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:23.964737 2214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:23.965950 2214 hybrid_clock.cc:648] HybridClock initialized: now 1790492063965917 us; error 45 us; skew 500 ppm
I20260927 06:54:23.966269 2214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:23 dist-test-slave-f892 krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:23.969167 2214 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:54:23.970639 2214 webserver.cc:460] Webserver started at http://127.0.97.1:39347/ using document root <none> and password file <none>
I20260927 06:54:23.971290 2214 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:23.971428 2214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:23.971736 2214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:23.974550 2214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "b45e2055eb8946c28e0324952d30d256"
format_stamp: "Formatted at 2026-09-27 06:54:23 on dist-test-slave-f892"
server_key: "a0acdf2ce1fbf399b342bff28828d4f2"
server_key_iv: "651af7fd6d1d8a38ac1cb9a47c6f237c"
server_key_version: "encryptionkey@0"
I20260927 06:54:23.975286 2214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b45e2055eb8946c28e0324952d30d256"
format_stamp: "Formatted at 2026-09-27 06:54:23 on dist-test-slave-f892"
server_key: "a0acdf2ce1fbf399b342bff28828d4f2"
server_key_iv: "651af7fd6d1d8a38ac1cb9a47c6f237c"
server_key_version: "encryptionkey@0"
I20260927 06:54:23.980227 2214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 06:54:23.983608 2229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:23.984782 2214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 06:54:23.984947 2214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "b45e2055eb8946c28e0324952d30d256"
format_stamp: "Formatted at 2026-09-27 06:54:23 on dist-test-slave-f892"
server_key: "a0acdf2ce1fbf399b342bff28828d4f2"
server_key_iv: "651af7fd6d1d8a38ac1cb9a47c6f237c"
server_key_version: "encryptionkey@0"
I20260927 06:54:23.985144 2214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:24.005580 2214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:24.009523 2214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:24.009794 2214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:24.012276 2214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:24.012390 2214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:24.012547 2214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:24.012684 2214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 27 06:54:24 dist-test-slave-f892 krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:24.032972 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:24.012785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53541 (local address 127.0.97.62:38755)
0927 06:54:24.012985 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:24.012991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:24.014354 (+ 1363us) server_negotiation.cc:407] Connection header received
0927 06:54:24.016263 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:24.016269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:24.016357 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:24.016465 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:24.018302 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.019292 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.021342 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.021957 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.022124 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:24.025953 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:24.025959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:24.025964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:24.026046 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:24.029213 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:24.030434 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:24.030438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:24.030440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:24.030564 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:24.031108 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:24.031111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:24.031113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:24.031447 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:24.031551 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:24.032592 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0927 06:54:24.032743 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 06:54:24.035044 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:24.013385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38755 (local address 127.0.97.1:53541)
0927 06:54:24.014193 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:24.014230 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:24.016002 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:24.016679 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:24.016686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:24.017299 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:24.018138 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:24.018156 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.019472 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.019479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:24.021193 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:24.021212 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.022349 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.022356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:24.022528 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:24.023263 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:24.023288 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:24.025741 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:24.029443 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:24.029454 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:24.029474 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:24.030301 (+ 827us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:24.030679 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:24.030687 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:24.030693 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:24.030984 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:24.031592 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:24.031604 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:24.032136 (+ 532us) client_negotiation.cc:769] Sending connection context
0927 06:54:24.032513 (+ 377us) client_negotiation.cc:240] Negotiation successful
0927 06:54:24.033185 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":351,"threads_started":1}
I20260927 06:54:24.058182 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:24.044734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49153 (local address 127.0.97.62:38755)
0927 06:54:24.044881 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:24.044885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:24.045481 (+ 596us) server_negotiation.cc:407] Connection header received
0927 06:54:24.045662 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:24.045668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:24.045725 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:24.045821 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:24.046985 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.047991 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.049995 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.050579 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.050716 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:24.053205 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:24.053213 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:24.053220 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:24.053282 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:24.055742 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:24.056348 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:24.056355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:24.056359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:24.056471 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:24.056809 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:24.056816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:24.056819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:24.057050 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:24.057210 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:24.057567 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 06:54:24.057778 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 06:54:24.058308 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:24.044781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38755 (local address 127.0.97.1:49153)
0927 06:54:24.045381 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:24.045399 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:24.045547 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:24.045841 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:24.045845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:24.046376 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:24.046854 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:24.046864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.048137 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.048142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:24.049851 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:24.049862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.050991 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.050995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:24.051143 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:24.051878 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:24.051906 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:24.052977 (+ 1071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:24.055863 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:24.055870 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:24.055874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:24.056219 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:24.056576 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:24.056583 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:24.056589 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:24.056709 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:24.057166 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:24.057171 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:24.057342 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 06:54:24.057480 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 06:54:24.057781 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":353,"threads_started":1}
I20260927 06:54:24.070535 2214 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34607
I20260927 06:54:24.070633 2348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34607 every 8 connection(s)
I20260927 06:54:24.071884 2214 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:54:24.073877 2349 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38755
I20260927 06:54:24.074138 2349 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:24.074852 2349 heartbeater.cc:511] Master 127.0.97.62:38755 requested a full tablet report, sending...
I20260927 06:54:24.075079 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 2214
I20260927 06:54:24.075273 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:54:24.076089 388 external_mini_cluster.cc:1428] Setting key 8a86f506cbd1d9b3996895d8a202fed8
I20260927 06:54:24.077025 2143 ts_manager.cc:194] Registered new tserver with Master: b45e2055eb8946c28e0324952d30d256 (127.0.97.1:34607)
I20260927 06:54:24.078677 2143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53541
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 06:54:24.115409 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38755
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:24.422501 2353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:24.423080 2353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:24.423291 2353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:24.430729 2353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:24.430848 2353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:24.431003 2353 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 06:54:24.431085 2353 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 06:54:24.431205 2353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:54:24.440868 2353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:24.442109 2353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:24.443356 2353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:24.455651 2358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:24.455698 2361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:24.455724 2359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:24.456758 2353 server_base.cc:1034] running on GCE node
I20260927 06:54:24.457294 2353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:24.458143 2353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:24.459350 2353 hybrid_clock.cc:648] HybridClock initialized: now 1790492064459306 us; error 53 us; skew 500 ppm
I20260927 06:54:24.459657 2353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:24 dist-test-slave-f892 krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:24.462620 2353 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:54:24.463887 2353 webserver.cc:460] Webserver started at http://127.0.97.2:46219/ using document root <none> and password file <none>
I20260927 06:54:24.464534 2353 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:24.464628 2353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:24.464864 2353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:24.467558 2353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "d58eeb98dc6242de9b52cdadbd4fb93a"
format_stamp: "Formatted at 2026-09-27 06:54:24 on dist-test-slave-f892"
server_key: "7a5972c7771e060b723defffa98a775f"
server_key_iv: "447133c511d8f17aace28d90201b11f2"
server_key_version: "encryptionkey@0"
I20260927 06:54:24.468228 2353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d58eeb98dc6242de9b52cdadbd4fb93a"
format_stamp: "Formatted at 2026-09-27 06:54:24 on dist-test-slave-f892"
server_key: "7a5972c7771e060b723defffa98a775f"
server_key_iv: "447133c511d8f17aace28d90201b11f2"
server_key_version: "encryptionkey@0"
I20260927 06:54:24.472962 2353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 06:54:24.476266 2368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:24.477433 2353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 06:54:24.477546 2353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "d58eeb98dc6242de9b52cdadbd4fb93a"
format_stamp: "Formatted at 2026-09-27 06:54:24 on dist-test-slave-f892"
server_key: "7a5972c7771e060b723defffa98a775f"
server_key_iv: "447133c511d8f17aace28d90201b11f2"
server_key_version: "encryptionkey@0"
I20260927 06:54:24.477689 2353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:24.497895 2353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:24.501829 2353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:24.502149 2353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:24.504645 2353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:24.504756 2353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:24.504882 2353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:24.504977 2353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 06:54:24 dist-test-slave-f892 krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:24.527901 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:24.505312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45313 (local address 127.0.97.62:38755)
0927 06:54:24.505503 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:24.505508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:24.506946 (+ 1438us) server_negotiation.cc:407] Connection header received
0927 06:54:24.508975 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:24.508982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:24.509072 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:24.509207 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:24.511473 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.512483 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.515012 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.515616 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.515738 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:24.520390 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:24.520397 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:24.520403 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:24.520458 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:24.523936 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:24.524878 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:24.524886 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:24.524890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:24.525003 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:24.525657 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:24.525663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:24.525667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:24.526094 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:24.526271 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:24.527135 (+ 864us) server_negotiation.cc:299] Negotiation successful
0927 06:54:24.527432 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 06:54:24.528347 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:24.505873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38755 (local address 127.0.97.2:45313)
0927 06:54:24.506780 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:24.506831 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:24.508632 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:24.509535 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:24.509546 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:24.510327 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:24.511273 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:24.511308 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.512667 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.512674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:24.514875 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:24.514886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.515854 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.515864 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:24.516046 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:24.516830 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:24.516866 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:24.520117 (+ 3251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:24.524080 (+ 3963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:24.524086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:24.524107 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:24.524728 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:24.525115 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:24.525119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:24.525123 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:24.525495 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:24.526248 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:24.526258 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:24.526607 (+ 349us) client_negotiation.cc:769] Sending connection context
0927 06:54:24.526978 (+ 371us) client_negotiation.cc:240] Negotiation successful
0927 06:54:24.527329 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":437,"threads_started":1}
I20260927 06:54:24.549715 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:24.533530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38755 (local address 127.0.97.2:43927)
0927 06:54:24.534170 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:24.534185 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:24.534309 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:24.534631 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:24.534636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:24.535057 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:24.535304 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:24.535312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.536345 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.536354 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:24.538620 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:24.538637 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.539609 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.539616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:24.539868 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:24.540576 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:24.540606 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:24.541978 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:24.546479 (+ 4501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:24.546483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:24.546486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:24.547105 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:24.547558 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:24.547568 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:24.547574 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:24.547763 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:24.548376 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:24.548381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:24.548674 (+ 293us) client_negotiation.cc:769] Sending connection context
0927 06:54:24.548833 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 06:54:24.549168 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":342,"threads_started":1}
I20260927 06:54:24.550820 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:24.533459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43927 (local address 127.0.97.62:38755)
0927 06:54:24.533604 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:24.533608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:24.534255 (+ 647us) server_negotiation.cc:407] Connection header received
0927 06:54:24.534443 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:24.534448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:24.534509 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:24.534604 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:24.535433 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.536209 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.538774 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.539185 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.539272 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:24.542247 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:24.542255 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:24.542263 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:24.542300 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:24.546342 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:24.547230 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:24.547235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:24.547239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:24.547415 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:24.547906 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:24.547912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:24.547918 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:24.548254 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:24.548420 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:24.550090 (+ 1670us) server_negotiation.cc:299] Negotiation successful
0927 06:54:24.550364 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 06:54:24.557647 2353 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39591
I20260927 06:54:24.557722 2487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39591 every 8 connection(s)
I20260927 06:54:24.558983 2353 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:54:24.560951 2488 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38755
I20260927 06:54:24.561106 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 2353
I20260927 06:54:24.561189 2488 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:24.561273 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:54:24.561795 388 external_mini_cluster.cc:1428] Setting key 507358ed5d342c215817c5d583a05d75
I20260927 06:54:24.561897 2488 heartbeater.cc:511] Master 127.0.97.62:38755 requested a full tablet report, sending...
I20260927 06:54:24.563211 2142 ts_manager.cc:194] Registered new tserver with Master: d58eeb98dc6242de9b52cdadbd4fb93a (127.0.97.2:39591)
I20260927 06:54:24.563988 2142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45313
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 06:54:24.600531 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38755
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:24.909456 2492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:24.909713 2492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:24.909871 2492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:24.916986 2492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:24.917073 2492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:24.917205 2492 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 06:54:24.917303 2492 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 06:54:24.917426 2492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:54:24.927008 2492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:24.928153 2492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:24.929441 2492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:24.942245 2497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:24.942281 2498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:24.942739 2500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:24.943130 2492 server_base.cc:1034] running on GCE node
I20260927 06:54:24.943773 2492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:24.944633 2492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:24.945852 2492 hybrid_clock.cc:648] HybridClock initialized: now 1790492064945815 us; error 52 us; skew 500 ppm
I20260927 06:54:24.946188 2492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:24 dist-test-slave-f892 krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:24.949028 2492 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:54:24.950407 2492 webserver.cc:460] Webserver started at http://127.0.97.3:45443/ using document root <none> and password file <none>
I20260927 06:54:24.951045 2492 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:24.951184 2492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:24.951478 2492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:24.954440 2492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "201204a8accf4e50b1afcc5a8ff0d1e5"
format_stamp: "Formatted at 2026-09-27 06:54:24 on dist-test-slave-f892"
server_key: "f321ddf9214653d1596e2512e32224fb"
server_key_iv: "c059289392cc3e6f0d22350f6a22dbac"
server_key_version: "encryptionkey@0"
I20260927 06:54:24.955163 2492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "201204a8accf4e50b1afcc5a8ff0d1e5"
format_stamp: "Formatted at 2026-09-27 06:54:24 on dist-test-slave-f892"
server_key: "f321ddf9214653d1596e2512e32224fb"
server_key_iv: "c059289392cc3e6f0d22350f6a22dbac"
server_key_version: "encryptionkey@0"
I20260927 06:54:24.959812 2492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 06:54:24.962961 2507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:24.964234 2492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 06:54:24.964388 2492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "201204a8accf4e50b1afcc5a8ff0d1e5"
format_stamp: "Formatted at 2026-09-27 06:54:24 on dist-test-slave-f892"
server_key: "f321ddf9214653d1596e2512e32224fb"
server_key_iv: "c059289392cc3e6f0d22350f6a22dbac"
server_key_version: "encryptionkey@0"
I20260927 06:54:24.964604 2492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:24.981809 2492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:24.985644 2492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:24.985917 2492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:24.988503 2492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:24.988634 2492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:24.988806 2492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:24.988912 2492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 06:54:24 dist-test-slave-f892 krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:25.011904 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:24.989155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53253 (local address 127.0.97.62:38755)
0927 06:54:24.989357 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:24.989365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:24.990856 (+ 1491us) server_negotiation.cc:407] Connection header received
0927 06:54:24.993044 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:24.993052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:24.993150 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:24.993265 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:24.995015 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.996279 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.998955 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.999808 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.999968 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.004091 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:25.004095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:25.004098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:25.004129 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:25.007823 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:25.008962 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:25.008988 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:25.008993 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:25.009135 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:25.009714 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:25.009721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:25.009726 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:25.010161 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:25.010339 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:25.011316 (+ 977us) server_negotiation.cc:299] Negotiation successful
0927 06:54:25.011554 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 06:54:25.013365 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:24.989814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38755 (local address 127.0.97.3:53253)
0927 06:54:24.990701 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:24.990737 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:24.992777 (+ 2040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:24.993484 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:24.993492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:24.993941 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:24.994740 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:24.994772 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:24.996452 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:24.996459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:24.998795 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:24.998812 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.000015 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.000019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.000195 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.000862 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:25.000897 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:25.003845 (+ 2948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:25.008008 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:25.008021 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:25.008040 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:25.008805 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:25.009262 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:25.009267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:25.009271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:25.009556 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:25.010321 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:25.010334 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:25.010829 (+ 495us) client_negotiation.cc:769] Sending connection context
0927 06:54:25.011223 (+ 394us) client_negotiation.cc:240] Negotiation successful
0927 06:54:25.011748 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":372,"threads_started":1}
I20260927 06:54:25.036406 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.019680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40781 (local address 127.0.97.62:38755)
0927 06:54:25.019851 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:25.019858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:25.020647 (+ 789us) server_negotiation.cc:407] Connection header received
0927 06:54:25.020829 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:25.020832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:25.020890 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:25.020975 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:25.022486 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.023573 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.026260 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.026686 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.026785 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.029376 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:25.029384 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:25.029391 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:25.029432 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:25.033703 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:25.034451 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:25.034459 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:25.034465 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:25.034627 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:25.034992 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:25.034997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:25.035001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:25.035258 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:25.035424 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:25.035805 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 06:54:25.036020 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 06:54:25.036445 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.019782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38755 (local address 127.0.97.3:40781)
0927 06:54:25.020514 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:25.020545 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:25.020707 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:25.021033 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:25.021040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:25.021771 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:25.022292 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.022308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.023743 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.023750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.025927 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.025945 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.026885 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.026894 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.027149 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.027820 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:25.027850 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:25.029099 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:25.033836 (+ 4737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:25.033840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:25.033842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:25.034321 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:25.034738 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:25.034743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:25.034747 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:25.034869 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:25.035373 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:25.035379 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:25.035556 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 06:54:25.035690 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 06:54:25.035994 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":384,"threads_started":1}
I20260927 06:54:25.049566 2492 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45451
I20260927 06:54:25.049598 2626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45451 every 8 connection(s)
I20260927 06:54:25.050830 2492 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:54:25.052907 2627 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38755
I20260927 06:54:25.053118 2627 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:25.053743 2627 heartbeater.cc:511] Master 127.0.97.62:38755 requested a full tablet report, sending...
I20260927 06:54:25.055029 2142 ts_manager.cc:194] Registered new tserver with Master: 201204a8accf4e50b1afcc5a8ff0d1e5 (127.0.97.3:45451)
I20260927 06:54:25.055759 2142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53253
I20260927 06:54:25.056852 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 2492
I20260927 06:54:25.057022 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:54:25.057427 388 external_mini_cluster.cc:1428] Setting key d90bf7d30b6c79fb73440f38c9080ed1
I20260927 06:54:25.059940 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:54:25.081830 2349 heartbeater.cc:503] Master 127.0.97.62:38755 was elected leader, sending a full tablet report...
Sep 27 06:54:25 dist-test-slave-f892 krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 06:54:25 dist-test-slave-f892 krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:25.104095 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.091219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60606 (local address 127.0.97.62:38755)
0927 06:54:25.091438 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:25.091445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:25.091596 (+ 151us) server_negotiation.cc:407] Connection header received
0927 06:54:25.091729 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:25.091732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:25.091800 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:25.091900 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:25.093226 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.093901 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.095117 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.095521 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.095617 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.099322 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:25.099326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:25.099329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:25.099363 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:25.101860 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:25.102475 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:25.102479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:25.102482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:25.102595 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:25.102909 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:25.102912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:25.102914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:25.103161 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:25.103324 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:25.103611 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 06:54:25.103761 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 06:54:25.107972 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:54:25.114677 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:54:25.149188 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.125784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54204 (local address 127.0.97.2:39591)
0927 06:54:25.141767 (+ 15983us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:25.141778 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:25.141806 (+ 28us) server_negotiation.cc:407] Connection header received
0927 06:54:25.141905 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:25.141912 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:25.142176 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:25.142371 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:25.143130 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.144496 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.146653 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.148045 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.148175 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.148374 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:25.148726 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 06:54:25.148898 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12450,"thread_start_us":15685,"threads_started":1}
I20260927 06:54:25.149299 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.125636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39591 (local address 127.0.0.1:54204)
0927 06:54:25.137872 (+ 12236us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:25.137920 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:25.138894 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:25.142366 (+ 3472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:25.142378 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:25.142413 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:25.142891 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.142908 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.144618 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.144625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.146501 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.146512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.148251 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.148257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.148465 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.148531 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 06:54:25.148680 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 06:54:25.148867 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11933,"mutex_wait_us":48,"thread_start_us":17440,"threads_started":1}
I20260927 06:54:25.150388 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.125824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45451 (local address 127.0.0.1:53084)
0927 06:54:25.140739 (+ 14915us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:25.140757 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:25.140952 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:25.141972 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:25.141979 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:25.142033 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:25.142518 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.142541 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.144543 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.144551 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.146940 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.146954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.149138 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.149143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.149781 (+ 638us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.149811 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 06:54:25.149896 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 06:54:25.150042 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14779,"mutex_wait_us":27,"thread_start_us":11572,"threads_started":1}
I20260927 06:54:25.151557 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.138594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34607 (local address 127.0.0.1:47964)
0927 06:54:25.143321 (+ 4727us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:25.143340 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:25.143667 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:25.144366 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:25.144372 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:25.144406 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:25.145042 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.145054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.146857 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.146864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.148812 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.148825 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.150749 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.150756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.150913 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.150944 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 06:54:25.151045 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 06:54:25.151173 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4611,"mutex_wait_us":1,"thread_start_us":1878,"threads_started":1}
I20260927 06:54:25.152060 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.138677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47964 (local address 127.0.97.1:34607)
0927 06:54:25.142921 (+ 4244us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:25.142931 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:25.143455 (+ 524us) server_negotiation.cc:407] Connection header received
0927 06:54:25.143843 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:25.143850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:25.144039 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:25.144194 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:25.145245 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.146724 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.148981 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.150321 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.150436 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.150625 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:25.151164 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 06:54:25.151326 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4081,"thread_start_us":3941,"threads_started":1}
I20260927 06:54:25.153107 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 71c2316910794ca79e9c14bc203425d5 (DEFAULT_TABLE table=test-table [id=1b137e4f560840c3992678a37f0b62ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:25.154410 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c2316910794ca79e9c14bc203425d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:25.154924 2281 tablet_service.cc:1467] Processing CreateTablet for tablet 71c2316910794ca79e9c14bc203425d5 (DEFAULT_TABLE table=test-table [id=1b137e4f560840c3992678a37f0b62ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:25.155908 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c2316910794ca79e9c14bc203425d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:25.162159 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.126006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53084 (local address 127.0.97.3:45451)
0927 06:54:25.140042 (+ 14036us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:25.140064 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:25.140861 (+ 797us) server_negotiation.cc:407] Connection header received
0927 06:54:25.141179 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:25.141187 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:25.141469 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:25.141634 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:25.142804 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.144427 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.147166 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.148665 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.148781 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.149004 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:25.149980 (+ 976us) server_negotiation.cc:299] Negotiation successful
0927 06:54:25.150159 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13811,"thread_start_us":14038,"threads_started":1}
I20260927 06:54:25.165190 2646 tablet_bootstrap.cc:492] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a: Bootstrap starting.
I20260927 06:54:25.165429 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 71c2316910794ca79e9c14bc203425d5 (DEFAULT_TABLE table=test-table [id=1b137e4f560840c3992678a37f0b62ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:25.166980 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c2316910794ca79e9c14bc203425d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:25.168067 2647 tablet_bootstrap.cc:492] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256: Bootstrap starting.
I20260927 06:54:25.168366 2646 tablet_bootstrap.cc:654] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:25.169655 2646 log.cc:824] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:25.171738 2647 tablet_bootstrap.cc:654] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:25.173190 2647 log.cc:824] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:25.176254 2647 tablet_bootstrap.cc:492] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256: No bootstrap required, opened a new log
I20260927 06:54:25.176515 2647 ts_tablet_manager.cc:1397] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 06:54:25.176705 2646 tablet_bootstrap.cc:492] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a: No bootstrap required, opened a new log
I20260927 06:54:25.176975 2646 ts_tablet_manager.cc:1397] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260927 06:54:25.181246 2650 tablet_bootstrap.cc:492] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5: Bootstrap starting.
I20260927 06:54:25.184970 2650 tablet_bootstrap.cc:654] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:25.185357 2647 raft_consensus.cc:348] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } }
I20260927 06:54:25.185786 2647 raft_consensus.cc:374] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:25.185891 2647 raft_consensus.cc:729] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45e2055eb8946c28e0324952d30d256, State: Initialized, Role: FOLLOWER
I20260927 06:54:25.186276 2650 log.cc:824] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:25.186332 2646 raft_consensus.cc:348] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } }
I20260927 06:54:25.186725 2646 raft_consensus.cc:374] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:25.186554 2647 consensus_queue.cc:260] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } }
I20260927 06:54:25.186838 2646 raft_consensus.cc:729] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58eeb98dc6242de9b52cdadbd4fb93a, State: Initialized, Role: FOLLOWER
I20260927 06:54:25.187400 2646 consensus_queue.cc:260] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } }
I20260927 06:54:25.188431 2488 heartbeater.cc:503] Master 127.0.97.62:38755 was elected leader, sending a full tablet report...
I20260927 06:54:25.189584 2650 tablet_bootstrap.cc:492] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5: No bootstrap required, opened a new log
I20260927 06:54:25.189601 2646 ts_tablet_manager.cc:1428] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260927 06:54:25.189833 2650 ts_tablet_manager.cc:1397] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260927 06:54:25.192577 2647 ts_tablet_manager.cc:1428] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256: Time spent starting tablet: real 0.016s user 0.012s sys 0.001s
I20260927 06:54:25.197741 2650 raft_consensus.cc:348] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } }
I20260927 06:54:25.198065 2650 raft_consensus.cc:374] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:25.198158 2650 raft_consensus.cc:729] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201204a8accf4e50b1afcc5a8ff0d1e5, State: Initialized, Role: FOLLOWER
I20260927 06:54:25.198632 2650 consensus_queue.cc:260] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } }
I20260927 06:54:25.199512 2627 heartbeater.cc:503] Master 127.0.97.62:38755 was elected leader, sending a full tablet report...
I20260927 06:54:25.199903 2650 ts_tablet_manager.cc:1428] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260927 06:54:25.257063 2654 raft_consensus.cc:482] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:54:25.257411 2654 raft_consensus.cc:504] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } }
I20260927 06:54:25.259060 2654 leader_election.cc:288] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d58eeb98dc6242de9b52cdadbd4fb93a (127.0.97.2:39591), b45e2055eb8946c28e0324952d30d256 (127.0.97.1:34607)
I20260927 06:54:25.265280 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.259487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44721 (local address 127.0.97.2:39591)
0927 06:54:25.259671 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:25.259677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:25.259704 (+ 27us) server_negotiation.cc:407] Connection header received
0927 06:54:25.259985 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:25.259992 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:25.260068 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:25.260171 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:25.260616 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.261828 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.263617 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.264353 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.264510 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.264613 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:25.264892 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 06:54:25.265015 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 06:54:25.265283 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.259310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39591 (local address 127.0.97.3:44721)
0927 06:54:25.259620 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:25.259641 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:25.259795 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:25.260196 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:25.260199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:25.260224 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:25.260458 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.260469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.261976 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.261981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.263464 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.263476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.264519 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.264525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.264718 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.264741 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 06:54:25.264847 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 06:54:25.265003 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260927 06:54:25.265643 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.259444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40203 (local address 127.0.97.1:34607)
0927 06:54:25.259629 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:25.259635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:25.260339 (+ 704us) server_negotiation.cc:407] Connection header received
0927 06:54:25.260504 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:25.260512 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:25.260621 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:25.260688 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:25.261362 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.262302 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.264002 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.264763 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.264916 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.265018 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:25.265289 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 06:54:25.265411 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 06:54:25.265723 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.259394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34607 (local address 127.0.97.3:40203)
0927 06:54:25.260225 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:25.260244 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:25.260378 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:25.260762 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:25.260767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:25.260794 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:25.261186 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.261198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.262492 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.262497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.263855 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.263866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.264978 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.264984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.265136 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.265161 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 06:54:25.265268 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 06:54:25.265392 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":412,"threads_started":1}
I20260927 06:54:25.266570 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c2316910794ca79e9c14bc203425d5" candidate_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" is_pre_election: true
I20260927 06:54:25.266605 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c2316910794ca79e9c14bc203425d5" candidate_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45e2055eb8946c28e0324952d30d256" is_pre_election: true
I20260927 06:54:25.267058 2438 raft_consensus.cc:2454] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201204a8accf4e50b1afcc5a8ff0d1e5 in term 0.
I20260927 06:54:25.267102 2304 raft_consensus.cc:2454] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201204a8accf4e50b1afcc5a8ff0d1e5 in term 0.
I20260927 06:54:25.267709 2510 leader_election.cc:302] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201204a8accf4e50b1afcc5a8ff0d1e5, d58eeb98dc6242de9b52cdadbd4fb93a; no voters:
I20260927 06:54:25.268141 2654 raft_consensus.cc:2781] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:54:25.268285 2654 raft_consensus.cc:482] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:54:25.268388 2654 raft_consensus.cc:3037] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:25.270758 2654 raft_consensus.cc:504] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } }
I20260927 06:54:25.271351 2654 leader_election.cc:288] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [CANDIDATE]: Term 1 election: Requested vote from peers d58eeb98dc6242de9b52cdadbd4fb93a (127.0.97.2:39591), b45e2055eb8946c28e0324952d30d256 (127.0.97.1:34607)
I20260927 06:54:25.271775 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c2316910794ca79e9c14bc203425d5" candidate_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a"
I20260927 06:54:25.271799 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c2316910794ca79e9c14bc203425d5" candidate_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45e2055eb8946c28e0324952d30d256"
I20260927 06:54:25.271981 2438 raft_consensus.cc:3037] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:25.271976 2304 raft_consensus.cc:3037] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:25.275359 2438 raft_consensus.cc:2454] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201204a8accf4e50b1afcc5a8ff0d1e5 in term 1.
I20260927 06:54:25.275359 2304 raft_consensus.cc:2454] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201204a8accf4e50b1afcc5a8ff0d1e5 in term 1.
I20260927 06:54:25.275717 2508 leader_election.cc:302] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201204a8accf4e50b1afcc5a8ff0d1e5, b45e2055eb8946c28e0324952d30d256; no voters:
I20260927 06:54:25.275997 2654 raft_consensus.cc:2781] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:25.276427 2654 raft_consensus.cc:686] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 LEADER]: Becoming Leader. State: Replica: 201204a8accf4e50b1afcc5a8ff0d1e5, State: Running, Role: LEADER
I20260927 06:54:25.276866 2654 consensus_queue.cc:237] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } }
I20260927 06:54:25.281490 2142 catalog_manager.cc:5495] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 201204a8accf4e50b1afcc5a8ff0d1e5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } health_report { overall_health: UNKNOWN } } }
W20260927 06:54:25.302472 2628 tablet.cc:2369] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:54:25.310732 2489 tablet.cc:2369] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 06:54:25 dist-test-slave-f892 krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
W20260927 06:54:25.324515 2350 tablet.cc:2369] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:54:25.328696 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.316283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38755 (local address 127.0.0.1:60616)
0927 06:54:25.316882 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:25.316897 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:25.317021 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:25.317314 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:25.317317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:25.317928 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:25.318239 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.318248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.319159 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.319162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.320095 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.320104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.320739 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.320743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.320864 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.321522 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:25.321538 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:25.323712 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:25.326623 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:25.326635 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:25.326648 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:25.327058 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:25.327362 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:25.327365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:25.327367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:25.327491 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:25.327917 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:25.327922 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:25.328051 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 06:54:25.328179 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 06:54:25.328359 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":330,"threads_started":1}
I20260927 06:54:25.328769 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.316281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.0.97.62:38755)
0927 06:54:25.316459 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:25.316479 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:25.316991 (+ 512us) server_negotiation.cc:407] Connection header received
0927 06:54:25.317127 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:25.317130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:25.317188 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:25.317271 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:25.318355 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.319059 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.320201 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.320580 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.320666 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.323876 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:25.323880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:25.323883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:25.323909 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:25.326490 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:25.327160 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:25.327164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:25.327166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:25.327270 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:25.327583 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:25.327585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:25.327587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:25.327818 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:25.327917 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:25.328262 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 06:54:25.328515 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":65}
I20260927 06:54:25.333483 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60616:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 06:54:25.334501 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:54:25.340494 2418 tablet_service.cc:1467] Processing CreateTablet for tablet c6a4fd4ab27d4129b8aaf4bf11cb6e36 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ae44c5117d44beb92e801e9d626ea4f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 06:54:25.340489 2281 tablet_service.cc:1467] Processing CreateTablet for tablet c6a4fd4ab27d4129b8aaf4bf11cb6e36 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ae44c5117d44beb92e801e9d626ea4f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 06:54:25.340808 2557 tablet_service.cc:1467] Processing CreateTablet for tablet c6a4fd4ab27d4129b8aaf4bf11cb6e36 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ae44c5117d44beb92e801e9d626ea4f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 06:54:25.341511 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a4fd4ab27d4129b8aaf4bf11cb6e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:25.341535 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a4fd4ab27d4129b8aaf4bf11cb6e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:25.341742 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a4fd4ab27d4129b8aaf4bf11cb6e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:25.347307 2650 tablet_bootstrap.cc:492] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5: Bootstrap starting.
I20260927 06:54:25.348610 2650 tablet_bootstrap.cc:654] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:25.348632 2647 tablet_bootstrap.cc:492] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256: Bootstrap starting.
I20260927 06:54:25.349004 2646 tablet_bootstrap.cc:492] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a: Bootstrap starting.
I20260927 06:54:25.350230 2647 tablet_bootstrap.cc:654] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:25.350545 2646 tablet_bootstrap.cc:654] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:25.351212 2650 tablet_bootstrap.cc:492] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5: No bootstrap required, opened a new log
I20260927 06:54:25.351403 2650 ts_tablet_manager.cc:1397] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260927 06:54:25.352368 2647 tablet_bootstrap.cc:492] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256: No bootstrap required, opened a new log
I20260927 06:54:25.352566 2647 ts_tablet_manager.cc:1397] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 06:54:25.352464 2650 raft_consensus.cc:348] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:25.352775 2650 raft_consensus.cc:374] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:25.352888 2650 raft_consensus.cc:729] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201204a8accf4e50b1afcc5a8ff0d1e5, State: Initialized, Role: FOLLOWER
I20260927 06:54:25.353174 2650 consensus_queue.cc:260] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:25.353381 2647 raft_consensus.cc:348] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:25.353631 2647 raft_consensus.cc:374] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:25.353718 2647 raft_consensus.cc:729] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45e2055eb8946c28e0324952d30d256, State: Initialized, Role: FOLLOWER
I20260927 06:54:25.353658 2654 tablet_replica.cc:439] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } }
I20260927 06:54:25.353858 2650 ts_tablet_manager.cc:1428] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260927 06:54:25.353952 2654 tablet_replica.cc:442] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:54:25.353996 2647 consensus_queue.cc:260] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:25.355327 2647 ts_tablet_manager.cc:1428] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 06:54:25.355187 2652 tablet_replica.cc:439] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } }
I20260927 06:54:25.355595 2652 tablet_replica.cc:442] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:54:25.359392 2646 tablet_bootstrap.cc:492] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a: No bootstrap required, opened a new log
I20260927 06:54:25.359503 2646 ts_tablet_manager.cc:1397] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260927 06:54:25.359998 2646 raft_consensus.cc:348] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:25.360158 2646 raft_consensus.cc:374] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:25.360221 2646 raft_consensus.cc:729] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58eeb98dc6242de9b52cdadbd4fb93a, State: Initialized, Role: FOLLOWER
I20260927 06:54:25.360419 2646 consensus_queue.cc:260] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:25.361016 2646 ts_tablet_manager.cc:1428] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 06:54:25.360854 2653 tablet_replica.cc:439] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } }
I20260927 06:54:25.361238 2653 tablet_replica.cc:442] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:54:25.445509 2652 raft_consensus.cc:482] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:54:25.445778 2652 raft_consensus.cc:504] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:25.447293 2652 leader_election.cc:288] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d58eeb98dc6242de9b52cdadbd4fb93a (127.0.97.2:39591), 201204a8accf4e50b1afcc5a8ff0d1e5 (127.0.97.3:45451)
I20260927 06:54:25.454300 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.448558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45567 (local address 127.0.97.3:45451)
0927 06:54:25.448764 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:25.448770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:25.449165 (+ 395us) server_negotiation.cc:407] Connection header received
0927 06:54:25.449309 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:25.449313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:25.449380 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:25.449480 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:25.450136 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.450795 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.452600 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.453382 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.453519 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.453640 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:25.453908 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 06:54:25.454045 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 06:54:25.454341 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.447784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45451 (local address 127.0.97.1:45567)
0927 06:54:25.449051 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:25.449069 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:25.449195 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:25.449497 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:25.449501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:25.449529 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:25.449941 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.449949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.450937 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.450951 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.452446 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.452457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.453587 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.453593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.453749 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.453770 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 06:54:25.453886 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 06:54:25.454026 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1127,"thread_start_us":657,"threads_started":1}
I20260927 06:54:25.454680 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.447545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33037 (local address 127.0.97.2:39591)
0927 06:54:25.447794 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:25.447800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:25.448305 (+ 505us) server_negotiation.cc:407] Connection header received
0927 06:54:25.448912 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:25.448918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:25.449022 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:25.449099 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:25.449886 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.450686 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.452703 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.453720 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.453863 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.453954 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:25.454261 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 06:54:25.454374 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260927 06:54:25.454892 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.447434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39591 (local address 127.0.97.1:33037)
0927 06:54:25.448162 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:25.448212 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:25.448617 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:25.449346 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:25.449349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:25.449378 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:25.449713 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.449723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.450805 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.450813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.452569 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.452580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.453922 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.453928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.454114 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.454139 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 06:54:25.454237 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 06:54:25.454514 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":27,"thread_start_us":1341,"threads_started":1}
I20260927 06:54:25.455492 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6a4fd4ab27d4129b8aaf4bf11cb6e36" candidate_uuid: "b45e2055eb8946c28e0324952d30d256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" is_pre_election: true
I20260927 06:54:25.455881 2581 raft_consensus.cc:2454] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45e2055eb8946c28e0324952d30d256 in term 0.
I20260927 06:54:25.456413 2230 leader_election.cc:302] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201204a8accf4e50b1afcc5a8ff0d1e5, b45e2055eb8946c28e0324952d30d256; no voters:
I20260927 06:54:25.456537 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6a4fd4ab27d4129b8aaf4bf11cb6e36" candidate_uuid: "b45e2055eb8946c28e0324952d30d256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" is_pre_election: true
I20260927 06:54:25.456797 2652 raft_consensus.cc:2781] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:54:25.456786 2438 raft_consensus.cc:2454] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45e2055eb8946c28e0324952d30d256 in term 0.
I20260927 06:54:25.456924 2652 raft_consensus.cc:482] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:54:25.457055 2652 raft_consensus.cc:3037] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:25.459174 2652 raft_consensus.cc:504] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:25.459705 2652 leader_election.cc:288] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [CANDIDATE]: Term 1 election: Requested vote from peers d58eeb98dc6242de9b52cdadbd4fb93a (127.0.97.2:39591), 201204a8accf4e50b1afcc5a8ff0d1e5 (127.0.97.3:45451)
I20260927 06:54:25.460112 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6a4fd4ab27d4129b8aaf4bf11cb6e36" candidate_uuid: "b45e2055eb8946c28e0324952d30d256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a"
I20260927 06:54:25.460175 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6a4fd4ab27d4129b8aaf4bf11cb6e36" candidate_uuid: "b45e2055eb8946c28e0324952d30d256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5"
I20260927 06:54:25.460336 2438 raft_consensus.cc:3037] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:25.460376 2581 raft_consensus.cc:3037] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:25.462536 2438 raft_consensus.cc:2454] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45e2055eb8946c28e0324952d30d256 in term 1.
I20260927 06:54:25.462544 2581 raft_consensus.cc:2454] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45e2055eb8946c28e0324952d30d256 in term 1.
I20260927 06:54:25.462975 2232 leader_election.cc:302] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b45e2055eb8946c28e0324952d30d256, d58eeb98dc6242de9b52cdadbd4fb93a; no voters:
I20260927 06:54:25.463364 2652 raft_consensus.cc:2781] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:25.463733 2652 raft_consensus.cc:686] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [term 1 LEADER]: Becoming Leader. State: Replica: b45e2055eb8946c28e0324952d30d256, State: Running, Role: LEADER
I20260927 06:54:25.464202 2652 consensus_queue.cc:237] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:25.468451 2669 tablet_replica.cc:439] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256: TxnStatusTablet state changed. Reason: New leader b45e2055eb8946c28e0324952d30d256. Latest consensus state: current_term: 1 leader_uuid: "b45e2055eb8946c28e0324952d30d256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } }
I20260927 06:54:25.468794 2669 tablet_replica.cc:442] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256: This TxnStatusTablet replica's current role is: LEADER
I20260927 06:54:25.469429 2671 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 06:54:25.469345 2142 catalog_manager.cc:5495] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b45e2055eb8946c28e0324952d30d256 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b45e2055eb8946c28e0324952d30d256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } health_report { overall_health: UNKNOWN } } }
I20260927 06:54:25.469713 2671 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 06:54:25.510429 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.503953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34607 (local address 127.0.0.1:47972)
0927 06:54:25.504404 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:25.504424 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:25.504572 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:25.505090 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:25.505095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:25.505108 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 06:54:25.505311 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.505321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.506905 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.506908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.507832 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.507843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.508754 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.508758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.508899 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.508948 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 06:54:25.509958 (+ 1010us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 06:54:25.509981 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 06:54:25.510113 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 06:54:25.510220 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297}
I20260927 06:54:25.510543 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.504036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47972 (local address 127.0.97.1:34607)
0927 06:54:25.504221 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:25.504227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:25.504516 (+ 289us) server_negotiation.cc:407] Connection header received
0927 06:54:25.504777 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:25.504783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:25.504890 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:25.505047 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:54:25.505496 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.506767 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.507969 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.508605 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.508747 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.509143 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:54:25.509848 (+ 705us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:54:25.509991 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:25.510158 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 06:54:25.510239 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 06:54:25.517588 2581 raft_consensus.cc:1261] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 FOLLOWER]: Refusing update from remote peer b45e2055eb8946c28e0324952d30d256: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:54:25.517612 2438 raft_consensus.cc:1261] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 FOLLOWER]: Refusing update from remote peer b45e2055eb8946c28e0324952d30d256: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:54:25.518522 2669 consensus_queue.cc:1035] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:25.518818 2652 consensus_queue.cc:1035] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:25.527837 2654 tablet_replica.cc:439] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5: TxnStatusTablet state changed. Reason: New leader b45e2055eb8946c28e0324952d30d256. Latest consensus state: current_term: 1 leader_uuid: "b45e2055eb8946c28e0324952d30d256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } }
I20260927 06:54:25.528113 2654 tablet_replica.cc:442] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:54:25.530189 2674 mvcc.cc:205] Tried to move back new op lower bound from 7333855500351152128 to 7333855500145500160. Current Snapshot: MvccSnapshot[applied={T|T < 7333855500351152128}]
I20260927 06:54:25.531690 2653 tablet_replica.cc:439] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a: TxnStatusTablet state changed. Reason: New leader b45e2055eb8946c28e0324952d30d256. Latest consensus state: current_term: 1 leader_uuid: "b45e2055eb8946c28e0324952d30d256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } }
I20260927 06:54:25.531927 2653 tablet_replica.cc:442] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:54:25.534089 2677 mvcc.cc:205] Tried to move back new op lower bound from 7333855500351152128 to 7333855500145500160. Current Snapshot: MvccSnapshot[applied={T|T < 7333855500351152128}]
I20260927 06:54:25.535564 2652 tablet_replica.cc:439] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b45e2055eb8946c28e0324952d30d256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } }
I20260927 06:54:25.535931 2652 tablet_replica.cc:442] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256: This TxnStatusTablet replica's current role is: LEADER
I20260927 06:54:25.537375 2654 tablet_replica.cc:439] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b45e2055eb8946c28e0324952d30d256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } }
I20260927 06:54:25.537624 2654 tablet_replica.cc:442] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P 201204a8accf4e50b1afcc5a8ff0d1e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:54:25.537474 2652 tablet_replica.cc:439] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b45e2055eb8946c28e0324952d30d256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } }
I20260927 06:54:25.537776 2652 tablet_replica.cc:442] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P b45e2055eb8946c28e0324952d30d256: This TxnStatusTablet replica's current role is: LEADER
I20260927 06:54:25.542666 2653 tablet_replica.cc:439] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b45e2055eb8946c28e0324952d30d256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } } }
I20260927 06:54:25.542903 2653 tablet_replica.cc:442] T c6a4fd4ab27d4129b8aaf4bf11cb6e36 P d58eeb98dc6242de9b52cdadbd4fb93a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:54:25.556100 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.550497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53094 (local address 127.0.97.3:45451)
0927 06:54:25.550657 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:25.550663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:25.550825 (+ 162us) server_negotiation.cc:407] Connection header received
0927 06:54:25.551083 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:25.551087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:25.551155 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:25.551294 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:54:25.551612 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.552361 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.553917 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.554363 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.554505 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.554901 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:54:25.555534 (+ 633us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:54:25.555678 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:25.555754 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 06:54:25.555829 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 06:54:25.577970 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.565669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37667 (local address 127.0.97.1:34607)
0927 06:54:25.565893 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:25.565902 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:25.573143 (+ 7241us) server_negotiation.cc:407] Connection header received
0927 06:54:25.573351 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:25.573355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:25.573417 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:25.573507 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:54:25.573976 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.574767 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.576286 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.576715 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.576849 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.577246 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:54:25.577338 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:54:25.577431 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:25.577575 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 06:54:25.577664 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 06:54:25.578080 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.565634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34607 (local address 127.0.97.3:37667)
0927 06:54:25.573041 (+ 7407us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:25.573056 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:25.573214 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:25.573539 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:25.573543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:25.573557 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 06:54:25.573828 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.573835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.574886 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.574893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.576152 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.576162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.576877 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.576883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.577003 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.577049 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 06:54:25.577431 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 06:54:25.577443 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 06:54:25.577555 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 06:54:25.577693 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7241,"thread_start_us":7087,"threads_started":1}
I20260927 06:54:25.588083 2437 raft_consensus.cc:1261] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 FOLLOWER]: Refusing update from remote peer 201204a8accf4e50b1afcc5a8ff0d1e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:54:25.588173 2304 raft_consensus.cc:1261] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256 [term 1 FOLLOWER]: Refusing update from remote peer 201204a8accf4e50b1afcc5a8ff0d1e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:54:25.588790 2656 consensus_queue.cc:1035] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:25.589207 2654 consensus_queue.cc:1035] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:25.593734 2675 mvcc.cc:205] Tried to move back new op lower bound from 7333855500641275904 to 7333855499377377280. Current Snapshot: MvccSnapshot[applied={T|T < 7333855500641275904}]
I20260927 06:54:25.624127 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.617672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34837 (local address 127.0.97.3:45451)
0927 06:54:25.617839 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:25.617845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:25.618123 (+ 278us) server_negotiation.cc:407] Connection header received
0927 06:54:25.618359 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:25.618363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:25.618433 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:25.618522 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:54:25.619090 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.620011 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.622217 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.622623 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.622777 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.623324 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:54:25.623444 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:54:25.623545 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:25.623735 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 06:54:25.623816 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 06:54:25.624554 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:25.617305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45451 (local address 127.0.97.1:34837)
0927 06:54:25.617981 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:25.618049 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:25.618224 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:25.618563 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:25.618568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:25.618582 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 06:54:25.618926 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.618934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.620163 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.620169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.622079 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:25.622095 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:25.622818 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:25.622825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:25.623002 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:25.623069 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 06:54:25.623600 (+ 531us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 06:54:25.623609 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 06:54:25.623706 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 06:54:25.623844 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":542,"threads_started":1}
I20260927 06:54:25.663842 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:54:25.664129 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:54:25.668282 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3: Sending DeleteTablet for 3 replicas of tablet 71c2316910794ca79e9c14bc203425d5
I20260927 06:54:25.669348 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet 71c2316910794ca79e9c14bc203425d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54204
I20260927 06:54:25.670140 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet 71c2316910794ca79e9c14bc203425d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53084
I20260927 06:54:25.671782 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:54:25.672395 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:54:25.674317 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet 71c2316910794ca79e9c14bc203425d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47964
I20260927 06:54:25.675877 2696 tablet_replica.cc:330] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5: stopping tablet replica
I20260927 06:54:25.676362 2696 raft_consensus.cc:2229] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:54:25.677109 2696 raft_consensus.cc:2258] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:25.681592 2417 tablet_service.cc:1467] Processing CreateTablet for tablet 1740b207e3bc432bb868f900f65b7fc5 (DEFAULT_TABLE table=test-table [id=f6c4274770a1412499a3a6a5be4739d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:25.682186 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1740b207e3bc432bb868f900f65b7fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:25.683128 2696 ts_tablet_manager.cc:1905] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:25.687088 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 1740b207e3bc432bb868f900f65b7fc5 (DEFAULT_TABLE table=test-table [id=f6c4274770a1412499a3a6a5be4739d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:25.687608 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1740b207e3bc432bb868f900f65b7fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:25.690636 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 1740b207e3bc432bb868f900f65b7fc5 (DEFAULT_TABLE table=test-table [id=f6c4274770a1412499a3a6a5be4739d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:25.691030 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1740b207e3bc432bb868f900f65b7fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:25.700845 2696 ts_tablet_manager.cc:1918] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:54:25.701036 2646 tablet_bootstrap.cc:492] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a: Bootstrap starting.
I20260927 06:54:25.701059 2696 log.cc:1196] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/71c2316910794ca79e9c14bc203425d5
I20260927 06:54:25.701730 2696 ts_tablet_manager.cc:1939] T 71c2316910794ca79e9c14bc203425d5 P 201204a8accf4e50b1afcc5a8ff0d1e5: Deleting consensus metadata
I20260927 06:54:25.702864 2646 tablet_bootstrap.cc:654] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:25.704144 2127 catalog_manager.cc:4841] TS 201204a8accf4e50b1afcc5a8ff0d1e5 (127.0.97.3:45451): tablet 71c2316910794ca79e9c14bc203425d5 (table test-table [id=1b137e4f560840c3992678a37f0b62ff]) successfully deleted
I20260927 06:54:25.716957 2647 tablet_bootstrap.cc:492] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256: Bootstrap starting.
I20260927 06:54:25.718922 2647 tablet_bootstrap.cc:654] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:25.722868 2650 tablet_bootstrap.cc:492] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5: Bootstrap starting.
I20260927 06:54:25.724653 2650 tablet_bootstrap.cc:654] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:25.733592 2647 tablet_bootstrap.cc:492] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256: No bootstrap required, opened a new log
I20260927 06:54:25.733808 2647 ts_tablet_manager.cc:1397] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.009s
I20260927 06:54:25.734659 2647 raft_consensus.cc:348] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:25.735037 2647 raft_consensus.cc:374] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:25.735167 2647 raft_consensus.cc:729] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45e2055eb8946c28e0324952d30d256, State: Initialized, Role: FOLLOWER
I20260927 06:54:25.735765 2647 consensus_queue.cc:260] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:25.736495 2647 ts_tablet_manager.cc:1428] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 06:54:25.748888 2650 tablet_bootstrap.cc:492] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5: No bootstrap required, opened a new log
I20260927 06:54:25.749116 2650 ts_tablet_manager.cc:1397] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.009s
I20260927 06:54:25.749910 2650 raft_consensus.cc:348] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:25.750191 2650 raft_consensus.cc:374] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:25.750278 2650 raft_consensus.cc:729] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201204a8accf4e50b1afcc5a8ff0d1e5, State: Initialized, Role: FOLLOWER
I20260927 06:54:25.750531 2650 consensus_queue.cc:260] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:25.751964 2646 tablet_bootstrap.cc:492] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a: No bootstrap required, opened a new log
I20260927 06:54:25.752211 2646 ts_tablet_manager.cc:1397] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.002s
I20260927 06:54:25.754261 2646 raft_consensus.cc:348] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:25.754534 2646 raft_consensus.cc:374] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:25.754616 2646 raft_consensus.cc:729] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58eeb98dc6242de9b52cdadbd4fb93a, State: Initialized, Role: FOLLOWER
I20260927 06:54:25.754876 2646 consensus_queue.cc:260] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:25.755460 2646 ts_tablet_manager.cc:1428] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 06:54:25.756132 2697 tablet_replica.cc:330] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256: stopping tablet replica
I20260927 06:54:25.756546 2697 raft_consensus.cc:2229] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:54:25.756867 2697 raft_consensus.cc:2258] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:25.759606 2650 ts_tablet_manager.cc:1428] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20260927 06:54:25.762125 2695 tablet_replica.cc:330] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a: stopping tablet replica
I20260927 06:54:25.762516 2695 raft_consensus.cc:2229] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:54:25.763638 2697 ts_tablet_manager.cc:1905] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:25.763883 2695 raft_consensus.cc:2258] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:25.765678 2695 ts_tablet_manager.cc:1905] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:25.771733 2697 ts_tablet_manager.cc:1918] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:54:25.771962 2697 log.cc:1196] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/71c2316910794ca79e9c14bc203425d5
I20260927 06:54:25.772596 2697 ts_tablet_manager.cc:1939] T 71c2316910794ca79e9c14bc203425d5 P b45e2055eb8946c28e0324952d30d256: Deleting consensus metadata
I20260927 06:54:25.773058 2695 ts_tablet_manager.cc:1918] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:54:25.773216 2695 log.cc:1196] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/71c2316910794ca79e9c14bc203425d5
I20260927 06:54:25.773748 2695 ts_tablet_manager.cc:1939] T 71c2316910794ca79e9c14bc203425d5 P d58eeb98dc6242de9b52cdadbd4fb93a: Deleting consensus metadata
I20260927 06:54:25.774451 2127 catalog_manager.cc:4841] TS b45e2055eb8946c28e0324952d30d256 (127.0.97.1:34607): tablet 71c2316910794ca79e9c14bc203425d5 (table test-table [id=1b137e4f560840c3992678a37f0b62ff]) successfully deleted
I20260927 06:54:25.775311 2129 catalog_manager.cc:4841] TS d58eeb98dc6242de9b52cdadbd4fb93a (127.0.97.2:39591): tablet 71c2316910794ca79e9c14bc203425d5 (table test-table [id=1b137e4f560840c3992678a37f0b62ff]) successfully deleted
I20260927 06:54:25.782032 2654 raft_consensus.cc:482] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:54:25.782217 2654 raft_consensus.cc:504] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:25.783020 2654 leader_election.cc:288] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b45e2055eb8946c28e0324952d30d256 (127.0.97.1:34607), d58eeb98dc6242de9b52cdadbd4fb93a (127.0.97.2:39591)
I20260927 06:54:25.783514 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1740b207e3bc432bb868f900f65b7fc5" candidate_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45e2055eb8946c28e0324952d30d256" is_pre_election: true
I20260927 06:54:25.783672 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1740b207e3bc432bb868f900f65b7fc5" candidate_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" is_pre_election: true
I20260927 06:54:25.783793 2304 raft_consensus.cc:2454] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201204a8accf4e50b1afcc5a8ff0d1e5 in term 0.
I20260927 06:54:25.783905 2438 raft_consensus.cc:2454] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201204a8accf4e50b1afcc5a8ff0d1e5 in term 0.
I20260927 06:54:25.784251 2508 leader_election.cc:302] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201204a8accf4e50b1afcc5a8ff0d1e5, b45e2055eb8946c28e0324952d30d256; no voters:
I20260927 06:54:25.784585 2654 raft_consensus.cc:2781] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:54:25.784715 2654 raft_consensus.cc:482] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:54:25.784814 2654 raft_consensus.cc:3037] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:25.786940 2654 raft_consensus.cc:504] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:25.787873 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1740b207e3bc432bb868f900f65b7fc5" candidate_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a"
I20260927 06:54:25.788072 2438 raft_consensus.cc:3037] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:25.790448 2438 raft_consensus.cc:2454] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201204a8accf4e50b1afcc5a8ff0d1e5 in term 1.
I20260927 06:54:25.790874 2510 leader_election.cc:302] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201204a8accf4e50b1afcc5a8ff0d1e5, d58eeb98dc6242de9b52cdadbd4fb93a; no voters:
I20260927 06:54:25.790874 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1740b207e3bc432bb868f900f65b7fc5" candidate_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45e2055eb8946c28e0324952d30d256"
I20260927 06:54:25.791066 2304 raft_consensus.cc:3037] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:25.791179 2656 raft_consensus.cc:2781] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:25.791311 2656 raft_consensus.cc:686] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 LEADER]: Becoming Leader. State: Replica: 201204a8accf4e50b1afcc5a8ff0d1e5, State: Running, Role: LEADER
I20260927 06:54:25.791555 2656 consensus_queue.cc:237] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:25.794253 2304 raft_consensus.cc:2454] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201204a8accf4e50b1afcc5a8ff0d1e5 in term 1.
I20260927 06:54:25.795287 2654 leader_election.cc:288] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [CANDIDATE]: Term 1 election: Requested vote from peers b45e2055eb8946c28e0324952d30d256 (127.0.97.1:34607), d58eeb98dc6242de9b52cdadbd4fb93a (127.0.97.2:39591)
I20260927 06:54:25.797408 2142 catalog_manager.cc:5495] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 201204a8accf4e50b1afcc5a8ff0d1e5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } health_report { overall_health: HEALTHY } } }
I20260927 06:54:25.834292 2304 raft_consensus.cc:1261] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256 [term 1 FOLLOWER]: Refusing update from remote peer 201204a8accf4e50b1afcc5a8ff0d1e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:54:25.835181 2654 consensus_queue.cc:1035] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:25.837510 2438 raft_consensus.cc:1261] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 FOLLOWER]: Refusing update from remote peer 201204a8accf4e50b1afcc5a8ff0d1e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:54:25.838131 2654 consensus_queue.cc:1035] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 06:54:25.874390 2675 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 06:54:27.461714 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:54:27.461917 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:54:27.465533 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3: Sending DeleteTablet for 3 replicas of tablet 1740b207e3bc432bb868f900f65b7fc5
I20260927 06:54:27.466275 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet 1740b207e3bc432bb868f900f65b7fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47964
I20260927 06:54:27.466332 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet 1740b207e3bc432bb868f900f65b7fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54204
I20260927 06:54:27.466343 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet 1740b207e3bc432bb868f900f65b7fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53084
I20260927 06:54:27.486555 2730 tablet_replica.cc:330] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256: stopping tablet replica
I20260927 06:54:27.486832 2730 raft_consensus.cc:2229] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:54:27.487110 2730 raft_consensus.cc:2258] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:27.487673 2731 tablet_replica.cc:330] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a: stopping tablet replica
I20260927 06:54:27.487851 2732 tablet_replica.cc:330] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5: stopping tablet replica
I20260927 06:54:27.487874 2731 raft_consensus.cc:2229] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:54:27.488047 2732 raft_consensus.cc:2229] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:54:27.488103 2731 raft_consensus.cc:2258] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:27.488417 2732 raft_consensus.cc:2258] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:27.488960 2730 ts_tablet_manager.cc:1905] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:27.489266 2732 ts_tablet_manager.cc:1905] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:27.490993 2731 ts_tablet_manager.cc:1905] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:27.496287 2732 ts_tablet_manager.cc:1918] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 06:54:27.496418 2732 log.cc:1196] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/1740b207e3bc432bb868f900f65b7fc5
I20260927 06:54:27.496845 2732 ts_tablet_manager.cc:1939] T 1740b207e3bc432bb868f900f65b7fc5 P 201204a8accf4e50b1afcc5a8ff0d1e5: Deleting consensus metadata
I20260927 06:54:27.497913 2127 catalog_manager.cc:4841] TS 201204a8accf4e50b1afcc5a8ff0d1e5 (127.0.97.3:45451): tablet 1740b207e3bc432bb868f900f65b7fc5 (table test-table [id=f6c4274770a1412499a3a6a5be4739d9]) successfully deleted
I20260927 06:54:27.498270 2731 ts_tablet_manager.cc:1918] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 06:54:27.498456 2731 log.cc:1196] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/1740b207e3bc432bb868f900f65b7fc5
I20260927 06:54:27.498934 2731 ts_tablet_manager.cc:1939] T 1740b207e3bc432bb868f900f65b7fc5 P d58eeb98dc6242de9b52cdadbd4fb93a: Deleting consensus metadata
I20260927 06:54:27.499951 2129 catalog_manager.cc:4841] TS d58eeb98dc6242de9b52cdadbd4fb93a (127.0.97.2:39591): tablet 1740b207e3bc432bb868f900f65b7fc5 (table test-table [id=f6c4274770a1412499a3a6a5be4739d9]) successfully deleted
I20260927 06:54:27.501267 2730 ts_tablet_manager.cc:1918] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 06:54:27.501394 2730 log.cc:1196] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/1740b207e3bc432bb868f900f65b7fc5
I20260927 06:54:27.501797 2730 ts_tablet_manager.cc:1939] T 1740b207e3bc432bb868f900f65b7fc5 P b45e2055eb8946c28e0324952d30d256: Deleting consensus metadata
I20260927 06:54:27.502764 2127 catalog_manager.cc:4841] TS b45e2055eb8946c28e0324952d30d256 (127.0.97.1:34607): tablet 1740b207e3bc432bb868f900f65b7fc5 (table test-table [id=f6c4274770a1412499a3a6a5be4739d9]) successfully deleted
Sep 27 06:54:27 dist-test-slave-f892 krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492067, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 06:54:27 dist-test-slave-f892 krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492067, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:27.534443 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:27.515400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60624 (local address 127.0.97.62:38755)
0927 06:54:27.516239 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:27.516243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:27.516259 (+ 16us) server_negotiation.cc:407] Connection header received
0927 06:54:27.516300 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:27.516303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:27.516353 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:27.516529 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:27.517748 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:27.518556 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:27.519745 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:27.520136 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:27.520260 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:27.529436 (+ 9176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:27.529440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:27.529443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:27.529469 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:27.532175 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:27.532771 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:27.532775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:27.532777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:27.532910 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:27.533251 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:27.533255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:27.533257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:27.533526 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:27.533717 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:27.533983 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 06:54:27.534199 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":604,"threads_started":1}
I20260927 06:54:27.537745 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:54:27.538457 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:54:27.544914 2281 tablet_service.cc:1467] Processing CreateTablet for tablet be0486041c9b4a09bac99f70d1bdbb6a (DEFAULT_TABLE table=test-table [id=50aa27dd3d4847a69c1c7602b7ce5e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:27.544912 2418 tablet_service.cc:1467] Processing CreateTablet for tablet be0486041c9b4a09bac99f70d1bdbb6a (DEFAULT_TABLE table=test-table [id=50aa27dd3d4847a69c1c7602b7ce5e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:27.545328 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be0486041c9b4a09bac99f70d1bdbb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:27.545246 2557 tablet_service.cc:1467] Processing CreateTablet for tablet be0486041c9b4a09bac99f70d1bdbb6a (DEFAULT_TABLE table=test-table [id=50aa27dd3d4847a69c1c7602b7ce5e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:27.545349 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be0486041c9b4a09bac99f70d1bdbb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:27.545643 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be0486041c9b4a09bac99f70d1bdbb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:27.550308 2741 tablet_bootstrap.cc:492] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256: Bootstrap starting.
I20260927 06:54:27.551867 2742 tablet_bootstrap.cc:492] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a: Bootstrap starting.
I20260927 06:54:27.552213 2741 tablet_bootstrap.cc:654] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:27.552752 2743 tablet_bootstrap.cc:492] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5: Bootstrap starting.
I20260927 06:54:27.553768 2742 tablet_bootstrap.cc:654] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:27.554625 2743 tablet_bootstrap.cc:654] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:27.556268 2742 tablet_bootstrap.cc:492] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a: No bootstrap required, opened a new log
I20260927 06:54:27.556461 2742 ts_tablet_manager.cc:1397] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 06:54:27.560971 2741 tablet_bootstrap.cc:492] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256: No bootstrap required, opened a new log
I20260927 06:54:27.561089 2741 ts_tablet_manager.cc:1397] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260927 06:54:27.561712 2741 raft_consensus.cc:348] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:27.561942 2741 raft_consensus.cc:374] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:27.562037 2741 raft_consensus.cc:729] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45e2055eb8946c28e0324952d30d256, State: Initialized, Role: FOLLOWER
I20260927 06:54:27.562268 2741 consensus_queue.cc:260] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:27.562635 2743 tablet_bootstrap.cc:492] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5: No bootstrap required, opened a new log
I20260927 06:54:27.562763 2743 ts_tablet_manager.cc:1397] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260927 06:54:27.562855 2741 ts_tablet_manager.cc:1428] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 06:54:27.563120 2742 raft_consensus.cc:348] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:27.563344 2742 raft_consensus.cc:374] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:27.563422 2742 raft_consensus.cc:729] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58eeb98dc6242de9b52cdadbd4fb93a, State: Initialized, Role: FOLLOWER
I20260927 06:54:27.563458 2743 raft_consensus.cc:348] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:27.563648 2742 consensus_queue.cc:260] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:27.563766 2743 raft_consensus.cc:374] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:27.563870 2743 raft_consensus.cc:729] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201204a8accf4e50b1afcc5a8ff0d1e5, State: Initialized, Role: FOLLOWER
I20260927 06:54:27.564118 2743 consensus_queue.cc:260] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:27.566679 2743 ts_tablet_manager.cc:1428] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260927 06:54:27.574518 2742 ts_tablet_manager.cc:1428] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a: Time spent starting tablet: real 0.018s user 0.005s sys 0.008s
I20260927 06:54:27.701053 2723 raft_consensus.cc:482] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:54:27.701292 2723 raft_consensus.cc:504] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:27.702059 2723 leader_election.cc:288] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b45e2055eb8946c28e0324952d30d256 (127.0.97.1:34607), d58eeb98dc6242de9b52cdadbd4fb93a (127.0.97.2:39591)
I20260927 06:54:27.702598 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0486041c9b4a09bac99f70d1bdbb6a" candidate_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45e2055eb8946c28e0324952d30d256" is_pre_election: true
I20260927 06:54:27.702708 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0486041c9b4a09bac99f70d1bdbb6a" candidate_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" is_pre_election: true
I20260927 06:54:27.702841 2304 raft_consensus.cc:2454] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201204a8accf4e50b1afcc5a8ff0d1e5 in term 0.
I20260927 06:54:27.703017 2438 raft_consensus.cc:2454] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201204a8accf4e50b1afcc5a8ff0d1e5 in term 0.
I20260927 06:54:27.703346 2508 leader_election.cc:302] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201204a8accf4e50b1afcc5a8ff0d1e5, b45e2055eb8946c28e0324952d30d256; no voters:
I20260927 06:54:27.703679 2723 raft_consensus.cc:2781] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:54:27.703789 2723 raft_consensus.cc:482] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:54:27.703892 2723 raft_consensus.cc:3037] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:27.706065 2723 raft_consensus.cc:504] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:27.706565 2723 leader_election.cc:288] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [CANDIDATE]: Term 1 election: Requested vote from peers b45e2055eb8946c28e0324952d30d256 (127.0.97.1:34607), d58eeb98dc6242de9b52cdadbd4fb93a (127.0.97.2:39591)
I20260927 06:54:27.706935 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0486041c9b4a09bac99f70d1bdbb6a" candidate_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45e2055eb8946c28e0324952d30d256"
I20260927 06:54:27.707005 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0486041c9b4a09bac99f70d1bdbb6a" candidate_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a"
I20260927 06:54:27.707114 2304 raft_consensus.cc:3037] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:27.707216 2438 raft_consensus.cc:3037] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:27.709043 2304 raft_consensus.cc:2454] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201204a8accf4e50b1afcc5a8ff0d1e5 in term 1.
I20260927 06:54:27.709079 2438 raft_consensus.cc:2454] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201204a8accf4e50b1afcc5a8ff0d1e5 in term 1.
I20260927 06:54:27.709421 2508 leader_election.cc:302] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201204a8accf4e50b1afcc5a8ff0d1e5, b45e2055eb8946c28e0324952d30d256; no voters:
I20260927 06:54:27.709728 2723 raft_consensus.cc:2781] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:27.709895 2723 raft_consensus.cc:686] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 LEADER]: Becoming Leader. State: Replica: 201204a8accf4e50b1afcc5a8ff0d1e5, State: Running, Role: LEADER
I20260927 06:54:27.710192 2723 consensus_queue.cc:237] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:27.712603 2141 catalog_manager.cc:5495] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 201204a8accf4e50b1afcc5a8ff0d1e5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } health_report { overall_health: HEALTHY } } }
I20260927 06:54:27.759065 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:27.753662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53100 (local address 127.0.97.3:45451)
0927 06:54:27.754509 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:27.754513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:27.754531 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:54:27.754576 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:27.754580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:27.754634 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:27.754710 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:54:27.755055 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:27.756147 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:27.757420 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:27.757792 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:27.757876 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:27.758340 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:54:27.758437 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:54:27.758505 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:27.758633 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 06:54:27.758686 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":641,"threads_started":1}
I20260927 06:54:27.766297 2438 raft_consensus.cc:1261] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 FOLLOWER]: Refusing update from remote peer 201204a8accf4e50b1afcc5a8ff0d1e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:54:27.766340 2304 raft_consensus.cc:1261] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256 [term 1 FOLLOWER]: Refusing update from remote peer 201204a8accf4e50b1afcc5a8ff0d1e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:54:27.766790 2720 consensus_queue.cc:1035] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:27.767076 2723 consensus_queue.cc:1035] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:27.810122 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:54:27.810323 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:54:27.812889 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3: Sending DeleteTablet for 3 replicas of tablet be0486041c9b4a09bac99f70d1bdbb6a
I20260927 06:54:27.813642 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet be0486041c9b4a09bac99f70d1bdbb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47964
I20260927 06:54:27.813647 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet be0486041c9b4a09bac99f70d1bdbb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54204
I20260927 06:54:27.813896 2730 tablet_replica.cc:330] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256: stopping tablet replica
I20260927 06:54:27.814070 2731 tablet_replica.cc:330] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a: stopping tablet replica
I20260927 06:54:27.814165 2730 raft_consensus.cc:2229] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:54:27.814229 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet be0486041c9b4a09bac99f70d1bdbb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53084
I20260927 06:54:27.814288 2731 raft_consensus.cc:2229] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:54:27.814337 2730 raft_consensus.cc:2258] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:27.814484 2732 tablet_replica.cc:330] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5: stopping tablet replica
I20260927 06:54:27.814486 2731 raft_consensus.cc:2258] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:27.814662 2732 raft_consensus.cc:2229] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:54:27.814951 2732 raft_consensus.cc:2258] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:27.815164 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 06:54:27.815420 2730 ts_tablet_manager.cc:1905] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:27.815610 2731 ts_tablet_manager.cc:1905] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 06:54:27.815738 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:54:27.815850 2732 ts_tablet_manager.cc:1905] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:27.821506 2730 ts_tablet_manager.cc:1918] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:54:27.821645 2730 log.cc:1196] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/be0486041c9b4a09bac99f70d1bdbb6a
I20260927 06:54:27.822444 2730 ts_tablet_manager.cc:1939] T be0486041c9b4a09bac99f70d1bdbb6a P b45e2055eb8946c28e0324952d30d256: Deleting consensus metadata
I20260927 06:54:27.822387 2281 tablet_service.cc:1467] Processing CreateTablet for tablet cd10c742615a452db1902222709bd36b (DEFAULT_TABLE table=test-table [id=7edd9edf4304418f9d29cf5ac6456ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:27.822482 2418 tablet_service.cc:1467] Processing CreateTablet for tablet cd10c742615a452db1902222709bd36b (DEFAULT_TABLE table=test-table [id=7edd9edf4304418f9d29cf5ac6456ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:27.822582 2732 ts_tablet_manager.cc:1918] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:54:27.822739 2732 log.cc:1196] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/be0486041c9b4a09bac99f70d1bdbb6a
I20260927 06:54:27.822901 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd10c742615a452db1902222709bd36b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:27.822909 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd10c742615a452db1902222709bd36b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:27.823153 2732 ts_tablet_manager.cc:1939] T be0486041c9b4a09bac99f70d1bdbb6a P 201204a8accf4e50b1afcc5a8ff0d1e5: Deleting consensus metadata
I20260927 06:54:27.823963 2127 catalog_manager.cc:4841] TS b45e2055eb8946c28e0324952d30d256 (127.0.97.1:34607): tablet be0486041c9b4a09bac99f70d1bdbb6a (table test-table [id=50aa27dd3d4847a69c1c7602b7ce5e61]) successfully deleted
I20260927 06:54:27.824319 2127 catalog_manager.cc:4841] TS 201204a8accf4e50b1afcc5a8ff0d1e5 (127.0.97.3:45451): tablet be0486041c9b4a09bac99f70d1bdbb6a (table test-table [id=50aa27dd3d4847a69c1c7602b7ce5e61]) successfully deleted
I20260927 06:54:27.825133 2557 tablet_service.cc:1467] Processing CreateTablet for tablet cd10c742615a452db1902222709bd36b (DEFAULT_TABLE table=test-table [id=7edd9edf4304418f9d29cf5ac6456ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:27.825477 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd10c742615a452db1902222709bd36b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:27.827953 2741 tablet_bootstrap.cc:492] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256: Bootstrap starting.
I20260927 06:54:27.828783 2742 tablet_bootstrap.cc:492] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a: Bootstrap starting.
I20260927 06:54:27.828832 2731 ts_tablet_manager.cc:1918] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:54:27.828970 2731 log.cc:1196] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/be0486041c9b4a09bac99f70d1bdbb6a
I20260927 06:54:27.829401 2731 ts_tablet_manager.cc:1939] T be0486041c9b4a09bac99f70d1bdbb6a P d58eeb98dc6242de9b52cdadbd4fb93a: Deleting consensus metadata
I20260927 06:54:27.829984 2741 tablet_bootstrap.cc:654] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:27.831403 2129 catalog_manager.cc:4841] TS d58eeb98dc6242de9b52cdadbd4fb93a (127.0.97.2:39591): tablet be0486041c9b4a09bac99f70d1bdbb6a (table test-table [id=50aa27dd3d4847a69c1c7602b7ce5e61]) successfully deleted
I20260927 06:54:27.831641 2743 tablet_bootstrap.cc:492] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5: Bootstrap starting.
I20260927 06:54:27.831758 2742 tablet_bootstrap.cc:654] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:27.832647 2741 tablet_bootstrap.cc:492] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256: No bootstrap required, opened a new log
I20260927 06:54:27.832846 2741 ts_tablet_manager.cc:1397] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 06:54:27.833328 2743 tablet_bootstrap.cc:654] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:27.833647 2741 raft_consensus.cc:348] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:27.833941 2741 raft_consensus.cc:374] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:27.834052 2742 tablet_bootstrap.cc:492] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a: No bootstrap required, opened a new log
I20260927 06:54:27.834043 2741 raft_consensus.cc:729] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45e2055eb8946c28e0324952d30d256, State: Initialized, Role: FOLLOWER
I20260927 06:54:27.834225 2742 ts_tablet_manager.cc:1397] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 06:54:27.834314 2741 consensus_queue.cc:260] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:27.835120 2742 raft_consensus.cc:348] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:27.835485 2741 ts_tablet_manager.cc:1428] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 06:54:27.835475 2742 raft_consensus.cc:374] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:27.835573 2742 raft_consensus.cc:729] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58eeb98dc6242de9b52cdadbd4fb93a, State: Initialized, Role: FOLLOWER
I20260927 06:54:27.835861 2742 consensus_queue.cc:260] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:27.836769 2742 ts_tablet_manager.cc:1428] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 06:54:27.838063 2743 tablet_bootstrap.cc:492] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5: No bootstrap required, opened a new log
I20260927 06:54:27.838231 2743 ts_tablet_manager.cc:1397] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 06:54:27.839407 2743 raft_consensus.cc:348] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:27.839617 2743 raft_consensus.cc:374] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:27.839709 2743 raft_consensus.cc:729] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201204a8accf4e50b1afcc5a8ff0d1e5, State: Initialized, Role: FOLLOWER
I20260927 06:54:27.839927 2743 consensus_queue.cc:260] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:27.840405 2743 ts_tablet_manager.cc:1428] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 06:54:27.949523 2747 raft_consensus.cc:482] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:54:27.949784 2747 raft_consensus.cc:504] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:27.951225 2747 leader_election.cc:288] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b45e2055eb8946c28e0324952d30d256 (127.0.97.1:34607), 201204a8accf4e50b1afcc5a8ff0d1e5 (127.0.97.3:45451)
I20260927 06:54:27.959093 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:27.951536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34607 (local address 127.0.97.2:48469)
0927 06:54:27.952105 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:27.952130 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:27.952287 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:27.952754 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:27.952758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:27.952799 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:27.953189 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:27.953202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:27.954527 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:27.954531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:27.956471 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:27.956485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:27.957950 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:27.957954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:27.958147 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:27.958171 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 06:54:27.958245 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 06:54:27.958364 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":23,"thread_start_us":269,"threads_started":1}
I20260927 06:54:27.959167 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:27.951568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48469 (local address 127.0.97.1:34607)
0927 06:54:27.952175 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:27.952181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:27.952214 (+ 33us) server_negotiation.cc:407] Connection header received
0927 06:54:27.952456 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:27.952465 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:27.952567 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:27.952687 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:27.953369 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:27.954390 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:27.956666 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:27.957783 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:27.957912 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:27.958045 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:27.958303 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 06:54:27.958434 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":279,"threads_started":1}
I20260927 06:54:27.959391 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:27.952371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44989 (local address 127.0.97.3:45451)
0927 06:54:27.952648 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:27.952656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:27.953055 (+ 399us) server_negotiation.cc:407] Connection header received
0927 06:54:27.953267 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:27.953271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:27.953394 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:27.953495 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:27.954048 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:27.954912 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:27.957091 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:27.958243 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:27.958365 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:27.958456 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:27.959036 (+ 580us) server_negotiation.cc:299] Negotiation successful
0927 06:54:27.959165 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260927 06:54:27.959403 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:27.952270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45451 (local address 127.0.97.2:44989)
0927 06:54:27.952952 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:27.952967 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:27.953127 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:27.953513 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:27.953516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:27.953535 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:27.953878 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:27.953887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:27.955065 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:27.955069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:27.956949 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:27.956962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:27.958446 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:27.958452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:27.958633 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:27.958662 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 06:54:27.958764 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 06:54:27.958849 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":3,"thread_start_us":365,"threads_started":1}
I20260927 06:54:27.960248 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd10c742615a452db1902222709bd36b" candidate_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45e2055eb8946c28e0324952d30d256" is_pre_election: true
I20260927 06:54:27.960248 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd10c742615a452db1902222709bd36b" candidate_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" is_pre_election: true
I20260927 06:54:27.960484 2304 raft_consensus.cc:2454] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d58eeb98dc6242de9b52cdadbd4fb93a in term 0.
I20260927 06:54:27.960492 2581 raft_consensus.cc:2454] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d58eeb98dc6242de9b52cdadbd4fb93a in term 0.
I20260927 06:54:27.960965 2369 leader_election.cc:302] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b45e2055eb8946c28e0324952d30d256, d58eeb98dc6242de9b52cdadbd4fb93a; no voters:
I20260927 06:54:27.961329 2747 raft_consensus.cc:2781] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:54:27.961480 2747 raft_consensus.cc:482] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:54:27.961584 2747 raft_consensus.cc:3037] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:27.963608 2747 raft_consensus.cc:504] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:27.964152 2747 leader_election.cc:288] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [CANDIDATE]: Term 1 election: Requested vote from peers b45e2055eb8946c28e0324952d30d256 (127.0.97.1:34607), 201204a8accf4e50b1afcc5a8ff0d1e5 (127.0.97.3:45451)
I20260927 06:54:27.964602 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd10c742615a452db1902222709bd36b" candidate_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45e2055eb8946c28e0324952d30d256"
I20260927 06:54:27.964640 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd10c742615a452db1902222709bd36b" candidate_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5"
I20260927 06:54:27.964818 2581 raft_consensus.cc:3037] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:27.964818 2304 raft_consensus.cc:3037] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:27.967654 2304 raft_consensus.cc:2454] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d58eeb98dc6242de9b52cdadbd4fb93a in term 1.
I20260927 06:54:27.967654 2581 raft_consensus.cc:2454] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d58eeb98dc6242de9b52cdadbd4fb93a in term 1.
I20260927 06:54:27.968015 2369 leader_election.cc:302] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b45e2055eb8946c28e0324952d30d256, d58eeb98dc6242de9b52cdadbd4fb93a; no voters:
I20260927 06:54:27.968262 2747 raft_consensus.cc:2781] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:27.968679 2747 raft_consensus.cc:686] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 LEADER]: Becoming Leader. State: Replica: d58eeb98dc6242de9b52cdadbd4fb93a, State: Running, Role: LEADER
I20260927 06:54:27.969077 2747 consensus_queue.cc:237] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } }
I20260927 06:54:27.971961 2143 catalog_manager.cc:5495] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a reported cstate change: term changed from 0 to 1, leader changed from <none> to d58eeb98dc6242de9b52cdadbd4fb93a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d58eeb98dc6242de9b52cdadbd4fb93a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 } health_report { overall_health: UNKNOWN } } }
I20260927 06:54:28.003495 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:27.998136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54220 (local address 127.0.97.2:39591)
0927 06:54:27.998736 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:27.998740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:27.998757 (+ 17us) server_negotiation.cc:407] Connection header received
0927 06:54:27.998829 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:27.998832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:27.998899 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:27.999042 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:54:27.999347 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:28.000197 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:28.001394 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:28.001771 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:28.001857 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:28.002417 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:54:28.002981 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:54:28.003093 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:28.003186 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 06:54:28.003245 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":365,"threads_started":1}
I20260927 06:54:28.017550 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:28.012088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34029 (local address 127.0.97.1:34607)
0927 06:54:28.012245 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:28.012250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:28.012913 (+ 663us) server_negotiation.cc:407] Connection header received
0927 06:54:28.013031 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:28.013034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:28.013094 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:28.013204 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:54:28.014174 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:28.014931 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:28.016211 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:28.016612 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:28.016709 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:28.017049 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:54:28.017133 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:54:28.017196 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:28.017307 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 06:54:28.017363 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 06:54:28.017621 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:28.012108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34607 (local address 127.0.97.2:34029)
0927 06:54:28.012805 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:28.012830 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:28.012932 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:28.013223 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:28.013226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:28.013234 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 06:54:28.013532 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:28.013572 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:28.015043 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:28.015047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:28.016100 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:28.016111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:28.016750 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:28.016754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:28.016870 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:28.016914 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 06:54:28.017212 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 06:54:28.017219 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 06:54:28.017276 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 06:54:28.017368 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":372,"threads_started":1}
I20260927 06:54:28.025346 2304 raft_consensus.cc:1261] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256 [term 1 FOLLOWER]: Refusing update from remote peer d58eeb98dc6242de9b52cdadbd4fb93a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:54:28.025317 2581 raft_consensus.cc:1261] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 FOLLOWER]: Refusing update from remote peer d58eeb98dc6242de9b52cdadbd4fb93a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:54:28.025976 2761 consensus_queue.cc:1035] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [LEADER]: Connected to new peer: Peer: permanent_uuid: "201204a8accf4e50b1afcc5a8ff0d1e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:28.026369 2747 consensus_queue.cc:1035] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b45e2055eb8946c28e0324952d30d256" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:28.051544 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:28.046519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48991 (local address 127.0.97.2:39591)
0927 06:54:28.046708 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:28.046715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:28.046952 (+ 237us) server_negotiation.cc:407] Connection header received
0927 06:54:28.047170 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:28.047174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:28.047228 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:28.047296 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:54:28.047634 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:28.048299 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:28.049936 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:28.050477 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:28.050577 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:28.051000 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:54:28.051084 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:54:28.051147 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:28.051279 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 06:54:28.051333 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 06:54:28.051707 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:28.046334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39591 (local address 127.0.97.1:48991)
0927 06:54:28.046861 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:28.046875 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:28.047020 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:28.047326 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:28.047329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:28.047337 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 06:54:28.047523 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:28.047530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:28.048437 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:28.048442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:28.049815 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:28.049828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:28.050630 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:28.050636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:28.050810 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:28.050856 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 06:54:28.051175 (+ 319us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 06:54:28.051185 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 06:54:28.051256 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 06:54:28.051384 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":267,"threads_started":1}
W20260927 06:54:28.052811 2713 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 06:54:29.639902 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:54:29.640134 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:54:29.644129 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5b85fde79904c3bbbcf6182d8994fa3: Sending DeleteTablet for 3 replicas of tablet cd10c742615a452db1902222709bd36b
I20260927 06:54:29.644815 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet cd10c742615a452db1902222709bd36b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47964
I20260927 06:54:29.644897 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet cd10c742615a452db1902222709bd36b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54204
I20260927 06:54:29.644908 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet cd10c742615a452db1902222709bd36b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53084
I20260927 06:54:29.645587 2795 tablet_replica.cc:330] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a: stopping tablet replica
I20260927 06:54:29.645889 2795 raft_consensus.cc:2229] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:54:29.645959 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 2214
I20260927 06:54:29.646672 2795 raft_consensus.cc:2258] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:29.647298 2794 tablet_replica.cc:330] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256: stopping tablet replica
I20260927 06:54:29.647509 2794 raft_consensus.cc:2229] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:54:29.647725 2794 raft_consensus.cc:2258] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:29.648725 2794 ts_tablet_manager.cc:1905] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:29.649693 2795 ts_tablet_manager.cc:1905] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:29.655318 2794 ts_tablet_manager.cc:1918] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 06:54:29.655494 2794 log.cc:1196] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/cd10c742615a452db1902222709bd36b
I20260927 06:54:29.655983 2794 ts_tablet_manager.cc:1939] T cd10c742615a452db1902222709bd36b P b45e2055eb8946c28e0324952d30d256: Deleting consensus metadata
I20260927 06:54:29.656090 2795 ts_tablet_manager.cc:1918] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 06:54:29.656220 2795 log.cc:1196] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/cd10c742615a452db1902222709bd36b
I20260927 06:54:29.656603 2795 ts_tablet_manager.cc:1939] T cd10c742615a452db1902222709bd36b P d58eeb98dc6242de9b52cdadbd4fb93a: Deleting consensus metadata
I20260927 06:54:29.657526 2129 catalog_manager.cc:4841] TS d58eeb98dc6242de9b52cdadbd4fb93a (127.0.97.2:39591): tablet cd10c742615a452db1902222709bd36b (table test-table [id=7edd9edf4304418f9d29cf5ac6456ca1]) successfully deleted
I20260927 06:54:29.657668 2796 tablet_replica.cc:330] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5: stopping tablet replica
I20260927 06:54:29.657846 2796 raft_consensus.cc:2229] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:54:29.658092 2796 raft_consensus.cc:2258] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:29.658650 2127 catalog_manager.cc:4841] TS b45e2055eb8946c28e0324952d30d256 (127.0.97.1:34607): tablet cd10c742615a452db1902222709bd36b (table test-table [id=7edd9edf4304418f9d29cf5ac6456ca1]) successfully deleted
I20260927 06:54:29.659299 2796 ts_tablet_manager.cc:1905] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:29.663758 2796 ts_tablet_manager.cc:1918] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 06:54:29.663918 2796 log.cc:1196] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/cd10c742615a452db1902222709bd36b
I20260927 06:54:29.664274 2796 ts_tablet_manager.cc:1939] T cd10c742615a452db1902222709bd36b P 201204a8accf4e50b1afcc5a8ff0d1e5: Deleting consensus metadata
I20260927 06:54:29.665323 2127 catalog_manager.cc:4841] TS 201204a8accf4e50b1afcc5a8ff0d1e5 (127.0.97.3:45451): tablet cd10c742615a452db1902222709bd36b (table test-table [id=7edd9edf4304418f9d29cf5ac6456ca1]) successfully deleted
Sep 27 06:54:29 dist-test-slave-f892 krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492067, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:54:29.682884 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:29.648665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.0.97.1:34607)
0927 06:54:29.654872 (+ 6207us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:29.654882 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:29.658715 (+ 3833us) server_negotiation.cc:407] Connection header received
0927 06:54:29.658865 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:29.658870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:29.658946 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:29.659061 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:29.660286 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:29.661022 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:29.662342 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:29.662762 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:29.662900 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:29.667803 (+ 4903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:29.667818 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:29.667835 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:29.667870 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:29.677992 (+ 10122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:29.678841 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:29.678850 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:29.678852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:29.678994 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:29.679381 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:29.679390 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:29.679393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:29.679744 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:29.680042 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:29.680322 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 06:54:29.682490 (+ 2168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6038,"thread_start_us":5925,"threads_started":1}
W20260927 06:54:29.683611 2344 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52290
W20260927 06:54:29.684319 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 2214: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:54:29.684553 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 2214
I20260927 06:54:29.706375 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 2353
Sep 27 06:54:29 dist-test-slave-f892 krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492067, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:54:29.720938 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:29.707104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.0.97.2:39591)
0927 06:54:29.707583 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:29.707588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:29.707604 (+ 16us) server_negotiation.cc:407] Connection header received
0927 06:54:29.707654 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:29.707657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:29.707719 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:29.707812 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:29.708947 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:29.709643 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:29.710864 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:29.711337 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:29.711433 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:29.715474 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:29.715487 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:29.715503 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:29.715532 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:29.718466 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:29.719110 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:29.719117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:29.719119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:29.719261 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:29.719573 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:29.719576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:29.719578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:29.719940 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:29.720143 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:29.720377 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 06:54:29.720545 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":282,"threads_started":1}
W20260927 06:54:29.721578 2483 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44948
W20260927 06:54:29.722286 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 2353: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:54:29.722440 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 2353
I20260927 06:54:29.744119 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 2492
Sep 27 06:54:29 dist-test-slave-f892 krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492067, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 06:54:29.758790 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:29.744835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50492 (local address 127.0.97.3:45451)
0927 06:54:29.745399 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:29.745404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:29.745420 (+ 16us) server_negotiation.cc:407] Connection header received
0927 06:54:29.745472 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:29.745475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:29.745538 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:29.745634 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:29.746948 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:29.747821 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:29.748934 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:29.749303 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:29.749388 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:29.753673 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:29.753686 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:29.753703 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:29.753745 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:29.756598 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:29.757156 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:29.757162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:29.757164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:29.757278 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:29.757556 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:29.757560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:29.757561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:29.757845 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:29.758111 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:29.758395 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 06:54:29.758540 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":316,"threads_started":1}
W20260927 06:54:29.759466 2622 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50492
W20260927 06:54:29.760072 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 2492: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:54:29.760243 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 2492
I20260927 06:54:29.784085 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 2111
I20260927 06:54:29.785290 2183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:29.903038 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 2111
2026-09-27T06:54:29Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8056 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:54:30 dist-test-slave-f892 krb5kdc[2804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:54:30 dist-test-slave-f892 krb5kdc[2804](info): set up 2 sockets
Sep 27 06:54:30 dist-test-slave-f892 krb5kdc[2804](info): commencing operation
krb5kdc: starting...
W20260927 06:54:31.026269 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.004s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:54:31 dist-test-slave-f892 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:54:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:54:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:54:31.217159 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43569
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:31.545540 2817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:31.545830 2817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:31.545979 2817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:31.553167 2817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:54:31.553272 2817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:31.553378 2817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:31.553475 2817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:54:31.553567 2817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:54:31.563215 2817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:31.564476 2817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:31.566018 2817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:31.579092 2822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:31.579137 2823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:31.579137 2825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:31.579690 2817 server_base.cc:1034] running on GCE node
I20260927 06:54:31.580677 2817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:31.582115 2817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:31.583319 2817 hybrid_clock.cc:648] HybridClock initialized: now 1790492071583268 us; error 46 us; skew 500 ppm
I20260927 06:54:31.583704 2817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:31 dist-test-slave-f892 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:31.586921 2817 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:54:31.588461 2817 webserver.cc:460] Webserver started at http://127.0.97.62:35389/ using document root <none> and password file <none>
I20260927 06:54:31.589154 2817 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:31.589283 2817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:31.589617 2817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:31.592628 2817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "4109014a1c5c4d27a70307567933c073"
format_stamp: "Formatted at 2026-09-27 06:54:31 on dist-test-slave-f892"
server_key: "7298d47dcf236e6151c21dc2b4aa24a8"
server_key_iv: "f19fde60e10e94df1bb72cd24797783b"
server_key_version: "encryptionkey@0"
I20260927 06:54:31.593423 2817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "4109014a1c5c4d27a70307567933c073"
format_stamp: "Formatted at 2026-09-27 06:54:31 on dist-test-slave-f892"
server_key: "7298d47dcf236e6151c21dc2b4aa24a8"
server_key_iv: "f19fde60e10e94df1bb72cd24797783b"
server_key_version: "encryptionkey@0"
I20260927 06:54:31.598724 2817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 06:54:31.605054 2832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:31.607018 2817 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260927 06:54:31.607213 2817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "4109014a1c5c4d27a70307567933c073"
format_stamp: "Formatted at 2026-09-27 06:54:31 on dist-test-slave-f892"
server_key: "7298d47dcf236e6151c21dc2b4aa24a8"
server_key_iv: "f19fde60e10e94df1bb72cd24797783b"
server_key_version: "encryptionkey@0"
I20260927 06:54:31.607460 2817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:31.620281 2817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:31.624353 2817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:31.624663 2817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:31.642170 2817 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43569
I20260927 06:54:31.642166 2883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43569 every 8 connection(s)
I20260927 06:54:31.643543 2817 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:54:31.644969 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 2817
I20260927 06:54:31.645223 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:54:31.645810 388 external_mini_cluster.cc:1428] Setting key 58b2fe57e509444b7be837e89e800e82
I20260927 06:54:31.647727 2884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 06:54:31 dist-test-slave-f892 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:31.661033 2884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073: Bootstrap starting.
I20260927 06:54:31.665282 2884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:31.666970 2884 log.cc:824] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:31.670782 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:31.647864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.0.97.62:43569)
0927 06:54:31.648745 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:31.648762 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:31.648817 (+ 55us) server_negotiation.cc:407] Connection header received
0927 06:54:31.651038 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:31.651065 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:31.651572 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:31.652007 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:31.653260 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:31.655311 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:31.656518 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:31.657502 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:31.657683 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:31.661511 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:31.661521 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:31.661540 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:31.661581 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:31.665458 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:31.666382 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:31.666390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:31.666400 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:31.666653 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:31.667119 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:31.667124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:31.667127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:31.667960 (+ 833us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:31.668340 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:31.668887 (+ 547us) server_negotiation.cc:299] Negotiation successful
0927 06:54:31.669436 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":299,"threads_started":1}
I20260927 06:54:31.671365 2884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073: No bootstrap required, opened a new log
I20260927 06:54:31.688237 2884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4109014a1c5c4d27a70307567933c073" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43569 } }
I20260927 06:54:31.688676 2884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:31.688794 2884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4109014a1c5c4d27a70307567933c073, State: Initialized, Role: FOLLOWER
I20260927 06:54:31.689446 2884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4109014a1c5c4d27a70307567933c073" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43569 } }
I20260927 06:54:31.689785 2884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:54:31.690012 2884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:54:31.690235 2884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:31.692340 2884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4109014a1c5c4d27a70307567933c073" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43569 } }
I20260927 06:54:31.692869 2884 leader_election.cc:302] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4109014a1c5c4d27a70307567933c073; no voters:
I20260927 06:54:31.693450 2884 leader_election.cc:288] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:54:31.693605 2889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:31.694269 2889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073 [term 1 LEADER]: Becoming Leader. State: Replica: 4109014a1c5c4d27a70307567933c073, State: Running, Role: LEADER
I20260927 06:54:31.694799 2889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4109014a1c5c4d27a70307567933c073" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43569 } }
I20260927 06:54:31.695576 2884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:54:31.698135 2890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4109014a1c5c4d27a70307567933c073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4109014a1c5c4d27a70307567933c073" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43569 } } }
I20260927 06:54:31.698043 2891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4109014a1c5c4d27a70307567933c073. Latest consensus state: current_term: 1 leader_uuid: "4109014a1c5c4d27a70307567933c073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4109014a1c5c4d27a70307567933c073" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43569 } } }
I20260927 06:54:31.698402 2890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:31.698462 2891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:31.700587 2897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:54:31.706102 2897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:54:31.715790 2897 catalog_manager.cc:1347] Generated new cluster ID: ed499c9e67e54475b4bbf7ebbef7efad
I20260927 06:54:31.715917 2897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:54:31.735510 2897 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:54:31.736819 2897 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:54:31.742697 2897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4109014a1c5c4d27a70307567933c073: Generated new TSK 0
I20260927 06:54:31.743562 2897 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 06:54:31.807574 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43569
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:32.144647 2910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:32.144934 2910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:32.145100 2910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:32.152287 2910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:32.152398 2910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:32.152594 2910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:54:32.162772 2910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:32.164003 2910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:32.165320 2910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:32.183789 2915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:32.184108 2918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:32.184152 2916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:32.184718 2910 server_base.cc:1034] running on GCE node
I20260927 06:54:32.185379 2910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:32.186467 2910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:32.187698 2910 hybrid_clock.cc:648] HybridClock initialized: now 1790492072187663 us; error 40 us; skew 500 ppm
I20260927 06:54:32.188027 2910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:32 dist-test-slave-f892 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:32.191218 2910 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:54:32.192729 2910 webserver.cc:460] Webserver started at http://127.0.97.1:40769/ using document root <none> and password file <none>
I20260927 06:54:32.193416 2910 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:32.193557 2910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:32.193862 2910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:32.196851 2910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "4dff5211db174ed7999d33de1124d759"
format_stamp: "Formatted at 2026-09-27 06:54:32 on dist-test-slave-f892"
server_key: "07009d5884525a7577f84782bf5ab15c"
server_key_iv: "5ce2b1f7560c21903fd3b59730de173d"
server_key_version: "encryptionkey@0"
I20260927 06:54:32.197598 2910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4dff5211db174ed7999d33de1124d759"
format_stamp: "Formatted at 2026-09-27 06:54:32 on dist-test-slave-f892"
server_key: "07009d5884525a7577f84782bf5ab15c"
server_key_iv: "5ce2b1f7560c21903fd3b59730de173d"
server_key_version: "encryptionkey@0"
I20260927 06:54:32.203693 2910 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260927 06:54:32.206892 2925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:32.208257 2910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:54:32.208418 2910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "4dff5211db174ed7999d33de1124d759"
format_stamp: "Formatted at 2026-09-27 06:54:32 on dist-test-slave-f892"
server_key: "07009d5884525a7577f84782bf5ab15c"
server_key_iv: "5ce2b1f7560c21903fd3b59730de173d"
server_key_version: "encryptionkey@0"
I20260927 06:54:32.208604 2910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:32.238875 2910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:32.243046 2910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:32.243330 2910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:32.244218 2910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:32.245601 2910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:32.245698 2910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:32.245838 2910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:32.245915 2910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:32.284073 2910 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34107
I20260927 06:54:32.284168 3037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34107 every 8 connection(s)
I20260927 06:54:32.285391 2910 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:54:32.291451 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 2910
I20260927 06:54:32.291702 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:54:32.292469 388 external_mini_cluster.cc:1428] Setting key 2d2ab772ae78705f5dd26da895709b76
Sep 27 06:54:32 dist-test-slave-f892 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:32.305563 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:32.287968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54131 (local address 127.0.97.62:43569)
0927 06:54:32.288370 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:32.288375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:32.289393 (+ 1018us) server_negotiation.cc:407] Connection header received
0927 06:54:32.291441 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:32.291445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:32.291518 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:32.291676 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:32.293587 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:32.294378 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:32.295953 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:32.296329 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:32.296459 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:32.299825 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:32.299830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:32.299833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:32.299862 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:32.302668 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:32.303339 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:32.303346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:32.303349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:32.303464 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:32.303875 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:32.303882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:32.303885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:32.304138 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:32.304289 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:32.304966 (+ 677us) server_negotiation.cc:299] Negotiation successful
0927 06:54:32.305196 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":210,"threads_started":1}
I20260927 06:54:32.306628 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:32.288566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43569 (local address 127.0.97.1:54131)
0927 06:54:32.289229 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:32.289265 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:32.291109 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:32.292005 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:32.292027 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:32.292770 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:32.293418 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:32.293435 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:32.294521 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:32.294528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:32.295787 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:32.295832 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:32.296537 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:32.296546 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:32.296844 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:32.297530 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:32.297551 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:32.299626 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:32.302803 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:32.302811 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:32.302827 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:32.303224 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:32.303579 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:32.303585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:32.303589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:32.303769 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:32.304265 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:32.304272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:32.304607 (+ 335us) client_negotiation.cc:769] Sending connection context
0927 06:54:32.304878 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 06:54:32.305425 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":273,"threads_started":1}
I20260927 06:54:32.308204 3038 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43569
I20260927 06:54:32.308499 3038 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 06:54:32.309231 3038 heartbeater.cc:511] Master 127.0.97.62:43569 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 06:54:32.311615 2849 ts_manager.cc:194] Registered new tserver with Master: 4dff5211db174ed7999d33de1124d759 (127.0.97.1:34107)
I20260927 06:54:32.313318 2849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 06:54:32.333142 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43569
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:32.691025 3044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:32.691315 3044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:32.691409 3044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:32.698764 3044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:32.698863 3044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:32.698971 3044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:54:32.708976 3044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:32.710210 3044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:32.711587 3044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:32.724251 3049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:32.724319 3052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:32.724294 3050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:32.730544 3044 server_base.cc:1034] running on GCE node
I20260927 06:54:32.731137 3044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:32.732014 3044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:32.733225 3044 hybrid_clock.cc:648] HybridClock initialized: now 1790492072733187 us; error 53 us; skew 500 ppm
I20260927 06:54:32.733525 3044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:32 dist-test-slave-f892 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:32.739499 3044 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:54:32.741067 3044 webserver.cc:460] Webserver started at http://127.0.97.2:38113/ using document root <none> and password file <none>
I20260927 06:54:32.741827 3044 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:32.741973 3044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:32.742333 3044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:32.745219 3044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "2b813611060d4861918d0eb77a77caa5"
format_stamp: "Formatted at 2026-09-27 06:54:32 on dist-test-slave-f892"
server_key: "173fa9d71ae32ca6931541450e978ee0"
server_key_iv: "46a1c6238079fa78f5150146e579901f"
server_key_version: "encryptionkey@0"
I20260927 06:54:32.746191 3044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2b813611060d4861918d0eb77a77caa5"
format_stamp: "Formatted at 2026-09-27 06:54:32 on dist-test-slave-f892"
server_key: "173fa9d71ae32ca6931541450e978ee0"
server_key_iv: "46a1c6238079fa78f5150146e579901f"
server_key_version: "encryptionkey@0"
I20260927 06:54:32.751296 3044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 06:54:32.754647 3059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:32.755988 3044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 06:54:32.756114 3044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "2b813611060d4861918d0eb77a77caa5"
format_stamp: "Formatted at 2026-09-27 06:54:32 on dist-test-slave-f892"
server_key: "173fa9d71ae32ca6931541450e978ee0"
server_key_iv: "46a1c6238079fa78f5150146e579901f"
server_key_version: "encryptionkey@0"
I20260927 06:54:32.756273 3044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:32.789438 3044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:32.793874 3044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:32.794185 3044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:32.795178 3044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:32.796511 3044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:32.796584 3044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:32.796672 3044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:32.796715 3044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:32.829726 3044 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46207
I20260927 06:54:32.829773 3171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46207 every 8 connection(s)
I20260927 06:54:32.830950 3044 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:54:32.835646 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 3044
I20260927 06:54:32.835831 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:54:32.836241 388 external_mini_cluster.cc:1428] Setting key 3d1583fd30c9068cb93f6b6f24bda4ca
Sep 27 06:54:32 dist-test-slave-f892 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:32.851442 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:32.833296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59101 (local address 127.0.97.62:43569)
0927 06:54:32.833736 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:32.833743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:32.834726 (+ 983us) server_negotiation.cc:407] Connection header received
0927 06:54:32.836637 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:32.836644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:32.836733 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:32.836841 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:32.838700 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:32.839914 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:32.841607 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:32.842385 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:32.842535 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:32.846050 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:32.846058 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:32.846063 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:32.846104 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:32.848492 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:32.849163 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:32.849168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:32.849172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:32.849295 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:32.849696 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:32.849703 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:32.849708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:32.850103 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:32.850254 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:32.850857 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 06:54:32.851080 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":213,"threads_started":1}
I20260927 06:54:32.852289 3175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:32.833833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43569 (local address 127.0.97.2:59101)
0927 06:54:32.834569 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:32.834612 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:32.836387 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:32.837094 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:32.837112 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:32.837794 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:32.838532 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:32.838550 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:32.840065 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:32.840068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:32.841385 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:32.841426 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:32.842581 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:32.842586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:32.842772 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:32.843443 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:32.843465 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:32.845791 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:32.848632 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:32.848640 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:32.848655 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:32.849040 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:32.849401 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:32.849408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:32.849411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:32.849587 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:32.850221 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:32.850229 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:32.850527 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 06:54:32.850786 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 06:54:32.851273 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":323,"threads_started":1}
I20260927 06:54:32.853585 3172 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43569
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 06:54:32.853906 3172 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:32.854595 3172 heartbeater.cc:511] Master 127.0.97.62:43569 requested a full tablet report, sending...
I20260927 06:54:32.855960 2849 ts_manager.cc:194] Registered new tserver with Master: 2b813611060d4861918d0eb77a77caa5 (127.0.97.2:46207)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 06:54:32.856906 2849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 06:54:32.886554 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43569
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:33.225158 3178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:33.225505 3178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:33.225708 3178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:33.233265 3178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:33.233422 3178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:33.233595 3178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:54:33.243317 3178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:33.244558 3178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:33.245893 3178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:33.258649 3184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:33.258625 3186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:33.258725 3183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:33.259352 3178 server_base.cc:1034] running on GCE node
I20260927 06:54:33.259951 3178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:33.260802 3178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:33.262066 3178 hybrid_clock.cc:648] HybridClock initialized: now 1790492073261979 us; error 37 us; skew 500 ppm
I20260927 06:54:33.262367 3178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:33 dist-test-slave-f892 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:33.275672 3178 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:54:33.278337 3178 webserver.cc:460] Webserver started at http://127.0.97.3:46719/ using document root <none> and password file <none>
I20260927 06:54:33.279086 3178 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:33.279239 3178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:33.279570 3178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:33.282615 3178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "5c489163470047a9b3c09b46dc4ff17a"
format_stamp: "Formatted at 2026-09-27 06:54:33 on dist-test-slave-f892"
server_key: "bd326a78151e5cdfba95de90aa34c656"
server_key_iv: "00cf12b7bd28d8d22b305a38af33c61e"
server_key_version: "encryptionkey@0"
I20260927 06:54:33.283404 3178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5c489163470047a9b3c09b46dc4ff17a"
format_stamp: "Formatted at 2026-09-27 06:54:33 on dist-test-slave-f892"
server_key: "bd326a78151e5cdfba95de90aa34c656"
server_key_iv: "00cf12b7bd28d8d22b305a38af33c61e"
server_key_version: "encryptionkey@0"
I20260927 06:54:33.288815 3178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 06:54:33.292150 3193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:33.293386 3178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 06:54:33.293557 3178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "5c489163470047a9b3c09b46dc4ff17a"
format_stamp: "Formatted at 2026-09-27 06:54:33 on dist-test-slave-f892"
server_key: "bd326a78151e5cdfba95de90aa34c656"
server_key_iv: "00cf12b7bd28d8d22b305a38af33c61e"
server_key_version: "encryptionkey@0"
I20260927 06:54:33.293752 3178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:33.316740 3038 heartbeater.cc:503] Master 127.0.97.62:43569 was elected leader, sending a full tablet report...
I20260927 06:54:33.319067 3178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:33.322894 3178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:33.323172 3178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:33.324105 3178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:33.325449 3178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:33.325527 3178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:33.325665 3178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:33.325714 3178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:33.358156 3178 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42773
I20260927 06:54:33.358204 3305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42773 every 8 connection(s)
I20260927 06:54:33.359405 3178 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:54:33.372625 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 3178
I20260927 06:54:33.372820 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
Sep 27 06:54:33 dist-test-slave-f892 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:33.373232 388 external_mini_cluster.cc:1428] Setting key 971840523f3476f590bff4ba801eec7c
I20260927 06:54:33.385972 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:33.361831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33921 (local address 127.0.97.62:43569)
0927 06:54:33.362270 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:33.362275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:33.363200 (+ 925us) server_negotiation.cc:407] Connection header received
0927 06:54:33.364955 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:33.364960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:33.365272 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:33.365435 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:33.367150 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:33.367893 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:33.369425 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:33.369833 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:33.369968 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:33.373930 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:33.373935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:33.373938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:33.373969 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:33.382779 (+ 8810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:33.383596 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:33.383602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:33.383606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:33.383731 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:33.384185 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:33.384190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:33.384193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:33.384463 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:33.384622 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:33.385360 (+ 738us) server_negotiation.cc:299] Negotiation successful
0927 06:54:33.385584 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":248,"threads_started":1}
I20260927 06:54:33.387008 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:33.362358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43569 (local address 127.0.97.3:33921)
0927 06:54:33.363048 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:33.363085 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:33.364689 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:33.365709 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:33.365718 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:33.366415 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:33.366970 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:33.366989 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:33.368061 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:33.368064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:33.369274 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:33.369285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:33.370091 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:33.370098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:33.370321 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:33.370994 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:33.371021 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:33.373724 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:33.382934 (+ 9210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:33.382946 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:33.382965 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:33.383458 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:33.383851 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:33.383857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:33.383860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:33.384062 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:33.384603 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:33.384612 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:33.384982 (+ 370us) client_negotiation.cc:769] Sending connection context
0927 06:54:33.385269 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 06:54:33.385833 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":266,"threads_started":1}
I20260927 06:54:33.388381 3306 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43569
I20260927 06:54:33.388692 3306 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:33.389379 3306 heartbeater.cc:511] Master 127.0.97.62:43569 requested a full tablet report, sending...
I20260927 06:54:33.390734 2849 ts_manager.cc:194] Registered new tserver with Master: 5c489163470047a9b3c09b46dc4ff17a (127.0.97.3:42773)
I20260927 06:54:33.391534 2849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33921
I20260927 06:54:33.394775 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 06:54:33.423681 3308 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:54:33.421176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57776 (local address 127.0.97.62:43569)
0927 06:54:33.421414 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:33.421421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:33.421729 (+ 308us) server_negotiation.cc:407] Connection header received
0927 06:54:33.421870 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:33.421874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:33.421955 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:33.422121 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:33.423416 (+ 1295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57776: BlockingRecv error: recv got EOF from 127.0.0.1:57776 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 06:54:33.426889 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 2910
W20260927 06:54:33.429860 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 2910: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:34107: server requires authentication, but client does not have Kerberos credentials available
I20260927 06:54:33.430084 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 2910
I20260927 06:54:33.445399 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 3044
W20260927 06:54:33.447856 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 3044: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:46207: server requires authentication, but client does not have Kerberos credentials available
I20260927 06:54:33.448035 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 3044
I20260927 06:54:33.460122 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 3178
W20260927 06:54:33.462847 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 3178: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:42773: server requires authentication, but client does not have Kerberos credentials available
I20260927 06:54:33.462992 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 3178
I20260927 06:54:33.478479 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 2817
I20260927 06:54:33.479687 2879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:33.589708 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 2817
2026-09-27T06:54:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3683 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:54:33 dist-test-slave-f892 krb5kdc[3324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:54:33 dist-test-slave-f892 krb5kdc[3324](info): set up 2 sockets
Sep 27 06:54:33 dist-test-slave-f892 krb5kdc[3324](info): commencing operation
krb5kdc: starting...
W20260927 06:54:34.730438 388 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:54:34 dist-test-slave-f892 krb5kdc[3324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:54:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:54:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:54:34.927489 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44459
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:35.249163 3337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:35.249432 3337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:35.249610 3337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:35.256492 3337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:54:35.256598 3337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:35.256712 3337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:35.256798 3337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:54:35.256886 3337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:54:35.266435 3337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:35.267800 3337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:35.269188 3337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:35.281972 3342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:35.282548 3343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:35.282662 3345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:35.283452 3337 server_base.cc:1034] running on GCE node
I20260927 06:54:35.284250 3337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:35.285609 3337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:35.286824 3337 hybrid_clock.cc:648] HybridClock initialized: now 1790492075286775 us; error 46 us; skew 500 ppm
I20260927 06:54:35.287160 3337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:35 dist-test-slave-f892 krb5kdc[3324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:35.290306 3337 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:54:35.291706 3337 webserver.cc:460] Webserver started at http://127.0.97.62:33871/ using document root <none> and password file <none>
I20260927 06:54:35.292379 3337 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:35.292513 3337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:35.292797 3337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:35.295646 3337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "201054c8b60f4211ad3c288f924d48af"
format_stamp: "Formatted at 2026-09-27 06:54:35 on dist-test-slave-f892"
server_key: "e6e873777514da1dbb5f76164ee3b20c"
server_key_iv: "0d40975af3a73bb9caa426c30363d1b9"
server_key_version: "encryptionkey@0"
I20260927 06:54:35.296378 3337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "201054c8b60f4211ad3c288f924d48af"
format_stamp: "Formatted at 2026-09-27 06:54:35 on dist-test-slave-f892"
server_key: "e6e873777514da1dbb5f76164ee3b20c"
server_key_iv: "0d40975af3a73bb9caa426c30363d1b9"
server_key_version: "encryptionkey@0"
I20260927 06:54:35.301096 3337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260927 06:54:35.304253 3352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:35.305534 3337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 06:54:35.305711 3337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "201054c8b60f4211ad3c288f924d48af"
format_stamp: "Formatted at 2026-09-27 06:54:35 on dist-test-slave-f892"
server_key: "e6e873777514da1dbb5f76164ee3b20c"
server_key_iv: "0d40975af3a73bb9caa426c30363d1b9"
server_key_version: "encryptionkey@0"
I20260927 06:54:35.305907 3337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:35.335464 3337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:35.339671 3337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:35.339926 3337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:35.357561 3337 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44459
I20260927 06:54:35.357599 3403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44459 every 8 connection(s)
I20260927 06:54:35.358892 3337 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:54:35.362414 3404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:35.364626 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 3337
I20260927 06:54:35.364825 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:54:35.365427 388 external_mini_cluster.cc:1428] Setting key ccc2595d5f3ef03791755c3c64c99826
I20260927 06:54:35.382164 3404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af: Bootstrap starting.
Sep 27 06:54:35 dist-test-slave-f892 krb5kdc[3324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:35.385840 3404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:35.386927 3404 log.cc:824] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:35.390196 3404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af: No bootstrap required, opened a new log
I20260927 06:54:35.391390 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:35.368306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.0.97.62:44459)
0927 06:54:35.369145 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:35.369157 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:35.369207 (+ 50us) server_negotiation.cc:407] Connection header received
0927 06:54:35.371008 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:35.371031 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:35.371468 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:35.371941 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:35.373499 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:35.375619 (+ 2120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:35.377794 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:35.378602 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:35.378861 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:35.383887 (+ 5026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:35.383899 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:35.383913 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:35.383976 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:35.387201 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:35.388042 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:35.388052 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:35.388063 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:35.388261 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:35.388686 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:35.388690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:35.388693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:35.389256 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:35.389469 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:35.389935 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 06:54:35.390392 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":375,"threads_started":1}
I20260927 06:54:35.395117 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:54:35.399603 3404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201054c8b60f4211ad3c288f924d48af" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44459 } }
I20260927 06:54:35.400032 3404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:35.400138 3404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201054c8b60f4211ad3c288f924d48af, State: Initialized, Role: FOLLOWER
I20260927 06:54:35.400779 3404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201054c8b60f4211ad3c288f924d48af" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44459 } }
I20260927 06:54:35.401053 3404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:54:35.401187 3404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:54:35.401350 3404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:35.403928 3404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201054c8b60f4211ad3c288f924d48af" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44459 } }
I20260927 06:54:35.404392 3404 leader_election.cc:302] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 201054c8b60f4211ad3c288f924d48af; no voters:
I20260927 06:54:35.405031 3404 leader_election.cc:288] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:54:35.405136 3410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:35.406433 3410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af [term 1 LEADER]: Becoming Leader. State: Replica: 201054c8b60f4211ad3c288f924d48af, State: Running, Role: LEADER
I20260927 06:54:35.406939 3410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201054c8b60f4211ad3c288f924d48af" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44459 } }
I20260927 06:54:35.407294 3404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:54:35.410269 3411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "201054c8b60f4211ad3c288f924d48af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201054c8b60f4211ad3c288f924d48af" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44459 } } }
I20260927 06:54:35.410619 3411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:35.409687 3412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 201054c8b60f4211ad3c288f924d48af. Latest consensus state: current_term: 1 leader_uuid: "201054c8b60f4211ad3c288f924d48af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201054c8b60f4211ad3c288f924d48af" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44459 } } }
I20260927 06:54:35.411106 3412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:35.414749 3416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:54:35.424773 3416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:54:35.444859 3416 catalog_manager.cc:1347] Generated new cluster ID: 9064bbaf4f81439e8bf8c95697ab1b4a
I20260927 06:54:35.444989 3416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:54:35.466161 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:35.457995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.0.97.62:44459)
0927 06:54:35.458209 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:35.458216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:35.459409 (+ 1193us) server_negotiation.cc:407] Connection header received
0927 06:54:35.459634 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:35.459639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:35.459706 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:35.459816 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:35.460788 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:35.461496 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:35.462816 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:35.463276 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:35.463407 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:35.464491 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:35.464497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:35.464502 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:35.465159 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:35.465222 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:35.465248 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:35.465371 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:35.465517 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 06:54:35.465628 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 06:54:35.473889 3416 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:54:35.475235 3416 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:54:35.485599 3416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 201054c8b60f4211ad3c288f924d48af: Generated new TSK 0
I20260927 06:54:35.486642 3416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:54:35.504536 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 3337
I20260927 06:54:35.505805 3399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:35.606380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 3337
2026-09-27T06:54:35Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2012 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:54:35 dist-test-slave-f892 krb5kdc[3439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:54:35 dist-test-slave-f892 krb5kdc[3439](info): set up 2 sockets
Sep 27 06:54:35 dist-test-slave-f892 krb5kdc[3439](info): commencing operation
krb5kdc: starting...
W20260927 06:54:36.730278 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:54:36 dist-test-slave-f892 krb5kdc[3439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:54:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:54:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:54:36.931108 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46301
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:37.262125 3452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:37.262452 3452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:37.262606 3452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:37.269850 3452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:54:37.270025 3452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:37.270160 3452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:37.270262 3452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:54:37.270344 3452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:54:37.283288 3452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:37.284698 3452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:37.285979 3452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:37.297236 3457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:37.297197 3458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:37.297271 3460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:37.298225 3452 server_base.cc:1034] running on GCE node
I20260927 06:54:37.298996 3452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:37.300352 3452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:37.301579 3452 hybrid_clock.cc:648] HybridClock initialized: now 1790492077301535 us; error 54 us; skew 500 ppm
I20260927 06:54:37.301908 3452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:37 dist-test-slave-f892 krb5kdc[3439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:37.304867 3452 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:54:37.306313 3452 webserver.cc:460] Webserver started at http://127.0.97.62:34273/ using document root <none> and password file <none>
I20260927 06:54:37.307005 3452 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:37.307126 3452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:37.307416 3452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:37.310350 3452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "2a2a402e5d9b4428a0143449d55c0936"
format_stamp: "Formatted at 2026-09-27 06:54:37 on dist-test-slave-f892"
server_key: "32c6c6ba8f489cea2a7c87a01ec8c9f3"
server_key_iv: "12292301d0d302689d1d0f3737d08faa"
server_key_version: "encryptionkey@0"
I20260927 06:54:37.311079 3452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "2a2a402e5d9b4428a0143449d55c0936"
format_stamp: "Formatted at 2026-09-27 06:54:37 on dist-test-slave-f892"
server_key: "32c6c6ba8f489cea2a7c87a01ec8c9f3"
server_key_iv: "12292301d0d302689d1d0f3737d08faa"
server_key_version: "encryptionkey@0"
I20260927 06:54:37.316001 3452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 06:54:37.319157 3467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:37.320315 3452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260927 06:54:37.320499 3452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "2a2a402e5d9b4428a0143449d55c0936"
format_stamp: "Formatted at 2026-09-27 06:54:37 on dist-test-slave-f892"
server_key: "32c6c6ba8f489cea2a7c87a01ec8c9f3"
server_key_iv: "12292301d0d302689d1d0f3737d08faa"
server_key_version: "encryptionkey@0"
I20260927 06:54:37.320672 3452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:37.337807 3452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:37.341281 3452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:37.341549 3452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:37.358959 3452 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46301
I20260927 06:54:37.358989 3518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46301 every 8 connection(s)
I20260927 06:54:37.360212 3452 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:54:37.363719 3519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:37.367692 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 3452
I20260927 06:54:37.367903 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:54:37.368527 388 external_mini_cluster.cc:1428] Setting key 18ecec90a562b6c00056ad8a34e2e3d9
I20260927 06:54:37.383531 3519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936: Bootstrap starting.
I20260927 06:54:37.386672 3519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:37.387719 3519 log.cc:824] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936: Log is configured to *not* fsync() on all Append() calls
Sep 27 06:54:37 dist-test-slave-f892 krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:37.390712 3519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936: No bootstrap required, opened a new log
I20260927 06:54:37.397053 3522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:37.373446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35408 (local address 127.0.97.62:46301)
0927 06:54:37.374201 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:37.374214 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:37.374259 (+ 45us) server_negotiation.cc:407] Connection header received
0927 06:54:37.377055 (+ 2796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:37.377081 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:37.377524 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:37.380940 (+ 3416us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:37.382635 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:37.384002 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:37.385241 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:37.385909 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:37.386141 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:37.389976 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:37.389984 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:37.389999 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:37.390089 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:37.393098 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:37.393799 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:37.393805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:37.393811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:37.393971 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:37.394424 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:37.394428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:37.394433 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:37.394945 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:37.395253 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:37.395681 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 06:54:37.396053 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":293,"threads_started":1}
I20260927 06:54:37.398674 3519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2a402e5d9b4428a0143449d55c0936" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46301 } }
I20260927 06:54:37.399036 3519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:37.399142 3519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a2a402e5d9b4428a0143449d55c0936, State: Initialized, Role: FOLLOWER
I20260927 06:54:37.399709 3519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2a402e5d9b4428a0143449d55c0936" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46301 } }
I20260927 06:54:37.399957 3519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:54:37.400096 3519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:54:37.400247 3519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:37.402593 3519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2a402e5d9b4428a0143449d55c0936" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46301 } }
I20260927 06:54:37.402997 3519 leader_election.cc:302] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a2a402e5d9b4428a0143449d55c0936; no voters:
I20260927 06:54:37.403465 3519 leader_election.cc:288] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:54:37.403622 3524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:37.404346 3524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936 [term 1 LEADER]: Becoming Leader. State: Replica: 2a2a402e5d9b4428a0143449d55c0936, State: Running, Role: LEADER
I20260927 06:54:37.404816 3524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2a402e5d9b4428a0143449d55c0936" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46301 } }
I20260927 06:54:37.405474 3519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:54:37.407886 3526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a2a402e5d9b4428a0143449d55c0936. Latest consensus state: current_term: 1 leader_uuid: "2a2a402e5d9b4428a0143449d55c0936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2a402e5d9b4428a0143449d55c0936" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46301 } } }
I20260927 06:54:37.408267 3526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:37.407872 3525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a2a402e5d9b4428a0143449d55c0936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2a402e5d9b4428a0143449d55c0936" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46301 } } }
I20260927 06:54:37.408537 3525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:37.414774 3529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:54:37.418848 3529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:54:37.426342 3529 catalog_manager.cc:1347] Generated new cluster ID: dfa3562bda614885bab9c2bb16362a94
I20260927 06:54:37.426461 3529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:54:37.440758 3529 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:54:37.442566 3529 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:54:37.451874 3529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a2a402e5d9b4428a0143449d55c0936: Generated new TSK 0
I20260927 06:54:37.452859 3529 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 06:54:37.511219 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46301
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:37.858119 3545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:37.858383 3545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:37.858556 3545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:37.865600 3545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:37.865720 3545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:37.865917 3545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:54:37.881446 3545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:37.882870 3545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:37.884227 3545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:37.900166 3550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:37.900154 3551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:37.900171 3553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:37.901051 3545 server_base.cc:1034] running on GCE node
I20260927 06:54:37.901757 3545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:37.902838 3545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:37.904106 3545 hybrid_clock.cc:648] HybridClock initialized: now 1790492077904051 us; error 74 us; skew 500 ppm
I20260927 06:54:37.904469 3545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:37 dist-test-slave-f892 krb5kdc[3439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:37.911828 3545 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:54:37.913546 3545 webserver.cc:460] Webserver started at http://127.0.97.1:39971/ using document root <none> and password file <none>
I20260927 06:54:37.914376 3545 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:37.914489 3545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:37.914753 3545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:37.917656 3545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "ee4195335bf54d289a79f2d817108600"
format_stamp: "Formatted at 2026-09-27 06:54:37 on dist-test-slave-f892"
server_key: "c1e46a32f122dc98a8fd78c7b6ddee1a"
server_key_iv: "0b614b52cee1a685f9d856f8f2afe407"
server_key_version: "encryptionkey@0"
I20260927 06:54:37.918458 3545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ee4195335bf54d289a79f2d817108600"
format_stamp: "Formatted at 2026-09-27 06:54:37 on dist-test-slave-f892"
server_key: "c1e46a32f122dc98a8fd78c7b6ddee1a"
server_key_iv: "0b614b52cee1a685f9d856f8f2afe407"
server_key_version: "encryptionkey@0"
I20260927 06:54:37.923636 3545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 06:54:37.927191 3560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:37.928642 3545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 06:54:37.928798 3545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "ee4195335bf54d289a79f2d817108600"
format_stamp: "Formatted at 2026-09-27 06:54:37 on dist-test-slave-f892"
server_key: "c1e46a32f122dc98a8fd78c7b6ddee1a"
server_key_iv: "0b614b52cee1a685f9d856f8f2afe407"
server_key_version: "encryptionkey@0"
I20260927 06:54:37.929011 3545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:37.946239 3545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:37.950555 3545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:37.950803 3545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:37.951808 3545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:37.953289 3545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:37.953361 3545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:37.953442 3545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:37.953495 3545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:38.001750 3545 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43185
I20260927 06:54:38.001798 3672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43185 every 8 connection(s)
I20260927 06:54:38.003041 3545 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:54:38.010851 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 3545
I20260927 06:54:38.011040 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:54:38.011633 388 external_mini_cluster.cc:1428] Setting key ebce4018db08f6b282d752ed9cf7c430
Sep 27 06:54:38 dist-test-slave-f892 krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:38.022744 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:38.005566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43517 (local address 127.0.97.62:46301)
0927 06:54:38.005990 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:38.005994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:38.007141 (+ 1147us) server_negotiation.cc:407] Connection header received
0927 06:54:38.008876 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:38.008881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:38.008964 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:38.009079 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:38.010554 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:38.011369 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:38.013206 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:38.013743 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:38.013919 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:38.017171 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:38.017176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:38.017179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:38.017207 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:38.019737 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:38.020613 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:38.020617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:38.020619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:38.020722 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:38.021174 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:38.021177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:38.021179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:38.021396 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:38.021508 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:38.022353 (+ 845us) server_negotiation.cc:299] Negotiation successful
0927 06:54:38.022511 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":209,"threads_started":1}
I20260927 06:54:38.024168 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:38.006202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46301 (local address 127.0.97.1:43517)
0927 06:54:38.006983 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:38.007024 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:38.008616 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:38.009305 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:38.009314 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:38.009780 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:38.010395 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:38.010411 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:38.011532 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:38.011536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:38.013029 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:38.013062 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:38.014035 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:38.014043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:38.014263 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:38.014926 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:38.014946 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:38.016967 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:38.019869 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:38.019881 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:38.019899 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:38.020497 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:38.020826 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:38.020834 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:38.020840 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:38.021057 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:38.021515 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:38.021526 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:38.021934 (+ 408us) client_negotiation.cc:769] Sending connection context
0927 06:54:38.022260 (+ 326us) client_negotiation.cc:240] Negotiation successful
0927 06:54:38.022783 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":283,"threads_started":1}
I20260927 06:54:38.025715 3673 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46301
I20260927 06:54:38.026064 3673 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:38.027063 3673 heartbeater.cc:511] Master 127.0.97.62:46301 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 06:54:38.029431 3484 ts_manager.cc:194] Registered new tserver with Master: ee4195335bf54d289a79f2d817108600 (127.0.97.1:43185)
I20260927 06:54:38.030997 3484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 06:54:38.054482 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46301
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:38.382771 3679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:38.383085 3679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:38.383179 3679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:38.390976 3679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:38.391111 3679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:38.391225 3679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:54:38.400996 3679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:38.402288 3679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:38.403543 3679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:38.416543 3684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:38.416752 3685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:38.416517 3687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:38.417320 3679 server_base.cc:1034] running on GCE node
I20260927 06:54:38.417932 3679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:38.418867 3679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:38.420058 3679 hybrid_clock.cc:648] HybridClock initialized: now 1790492078420013 us; error 31 us; skew 500 ppm
I20260927 06:54:38.420361 3679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:38 dist-test-slave-f892 krb5kdc[3439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:38.423360 3679 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:54:38.424787 3679 webserver.cc:460] Webserver started at http://127.0.97.2:46349/ using document root <none> and password file <none>
I20260927 06:54:38.425442 3679 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:38.425589 3679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:38.425879 3679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:38.428879 3679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "a4a270dc90264121a7d60b1033fe676e"
format_stamp: "Formatted at 2026-09-27 06:54:38 on dist-test-slave-f892"
server_key: "3c5ad9238e02621308f382a04fbd1ae4"
server_key_iv: "f7a70da68f793851318aff095e6a332e"
server_key_version: "encryptionkey@0"
I20260927 06:54:38.429695 3679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a4a270dc90264121a7d60b1033fe676e"
format_stamp: "Formatted at 2026-09-27 06:54:38 on dist-test-slave-f892"
server_key: "3c5ad9238e02621308f382a04fbd1ae4"
server_key_iv: "f7a70da68f793851318aff095e6a332e"
server_key_version: "encryptionkey@0"
I20260927 06:54:38.435053 3679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 06:54:38.438490 3694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:38.439831 3679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 06:54:38.439997 3679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "a4a270dc90264121a7d60b1033fe676e"
format_stamp: "Formatted at 2026-09-27 06:54:38 on dist-test-slave-f892"
server_key: "3c5ad9238e02621308f382a04fbd1ae4"
server_key_iv: "f7a70da68f793851318aff095e6a332e"
server_key_version: "encryptionkey@0"
I20260927 06:54:38.440179 3679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:38.488433 3679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:38.492741 3679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:38.493036 3679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:38.494050 3679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:38.495328 3679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:38.495440 3679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:38.495585 3679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:38.495656 3679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:38.528478 3679 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40029
I20260927 06:54:38.528527 3806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40029 every 8 connection(s)
I20260927 06:54:38.529711 3679 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:54:38.536669 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 3679
I20260927 06:54:38.536849 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:54:38.537317 388 external_mini_cluster.cc:1428] Setting key 1670f309a428483922d9a88a659730ce
Sep 27 06:54:38 dist-test-slave-f892 krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:38.550751 3809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:38.532087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33077 (local address 127.0.97.62:46301)
0927 06:54:38.532502 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:38.532506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:38.533505 (+ 999us) server_negotiation.cc:407] Connection header received
0927 06:54:38.535277 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:38.535283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:38.535368 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:38.535480 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:38.537105 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:38.537899 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:38.539975 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:38.540356 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:38.540447 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:38.544997 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:38.545006 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:38.545012 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:38.545052 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:38.547691 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:38.548461 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:38.548469 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:38.548472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:38.548596 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:38.549026 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:38.549034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:38.549037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:38.549294 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:38.549449 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:38.550164 (+ 715us) server_negotiation.cc:299] Negotiation successful
0927 06:54:38.550378 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":215,"threads_started":1}
I20260927 06:54:38.551729 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:38.532677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46301 (local address 127.0.97.2:33077)
0927 06:54:38.533363 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:38.533399 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:38.535032 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:38.535695 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:38.535702 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:38.536172 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:38.536929 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:38.536953 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:38.538141 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:38.538145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:38.539788 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:38.539802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:38.540571 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:38.540581 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:38.540902 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:38.541634 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:38.541665 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:38.544721 (+ 3056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:38.547851 (+ 3130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:38.547861 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:38.547879 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:38.548328 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:38.548709 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:38.548714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:38.548717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:38.548910 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:38.549427 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:38.549435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:38.549783 (+ 348us) client_negotiation.cc:769] Sending connection context
0927 06:54:38.550086 (+ 303us) client_negotiation.cc:240] Negotiation successful
0927 06:54:38.550637 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":262,"threads_started":1}
I20260927 06:54:38.553313 3807 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46301
I20260927 06:54:38.553757 3807 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:38.554551 3807 heartbeater.cc:511] Master 127.0.97.62:46301 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 06:54:38.556016 3484 ts_manager.cc:194] Registered new tserver with Master: a4a270dc90264121a7d60b1033fe676e (127.0.97.2:40029)
I20260927 06:54:38.556813 3484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 06:54:38.588769 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46301
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:38.928570 3813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:38.928830 3813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:38.928997 3813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:38.936684 3813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:38.936807 3813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:38.936990 3813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:54:38.946345 3813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:38.947604 3813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:38.948899 3813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:38.961670 3819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:38.961687 3818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:38.962098 3821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:38.962440 3813 server_base.cc:1034] running on GCE node
I20260927 06:54:38.963001 3813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:38.963836 3813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:38.965085 3813 hybrid_clock.cc:648] HybridClock initialized: now 1790492078965051 us; error 61 us; skew 500 ppm
I20260927 06:54:38.965394 3813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:38 dist-test-slave-f892 krb5kdc[3439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:38.968936 3813 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:54:38.972062 3813 webserver.cc:460] Webserver started at http://127.0.97.3:44531/ using document root <none> and password file <none>
I20260927 06:54:38.972751 3813 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:38.972895 3813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:38.973184 3813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:38.976073 3813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "01e3ad4b355149df82ed17438a478a68"
format_stamp: "Formatted at 2026-09-27 06:54:38 on dist-test-slave-f892"
server_key: "6e03e98d5038dbb87f1ef1a2e1f08eaa"
server_key_iv: "701b7f98534997056f4c7d5abed4d69c"
server_key_version: "encryptionkey@0"
I20260927 06:54:38.976848 3813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "01e3ad4b355149df82ed17438a478a68"
format_stamp: "Formatted at 2026-09-27 06:54:38 on dist-test-slave-f892"
server_key: "6e03e98d5038dbb87f1ef1a2e1f08eaa"
server_key_iv: "701b7f98534997056f4c7d5abed4d69c"
server_key_version: "encryptionkey@0"
I20260927 06:54:38.982195 3813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 06:54:38.986228 3828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:38.987763 3813 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 06:54:38.988067 3813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "01e3ad4b355149df82ed17438a478a68"
format_stamp: "Formatted at 2026-09-27 06:54:38 on dist-test-slave-f892"
server_key: "6e03e98d5038dbb87f1ef1a2e1f08eaa"
server_key_iv: "701b7f98534997056f4c7d5abed4d69c"
server_key_version: "encryptionkey@0"
I20260927 06:54:38.988293 3813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:39.007346 3813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:39.011369 3813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:39.011690 3813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:39.012665 3813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:39.014420 3813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:39.014528 3813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:39.014679 3813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:39.014750 3813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:39.034772 3673 heartbeater.cc:503] Master 127.0.97.62:46301 was elected leader, sending a full tablet report...
I20260927 06:54:39.047634 3813 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35353
I20260927 06:54:39.047715 3940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35353 every 8 connection(s)
I20260927 06:54:39.048813 3813 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:54:39.057679 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 3813
I20260927 06:54:39.057843 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:54:39.058389 388 external_mini_cluster.cc:1428] Setting key 4429c3a77a12f1925534db88cbdaa480
Sep 27 06:54:39 dist-test-slave-f892 krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:39.078606 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:39.051180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59301 (local address 127.0.97.62:46301)
0927 06:54:39.051626 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:39.051632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:39.052722 (+ 1090us) server_negotiation.cc:407] Connection header received
0927 06:54:39.054460 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:39.054468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:39.054561 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:39.054653 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:39.056029 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:39.056760 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:39.058856 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:39.059429 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:39.059568 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:39.062764 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:39.062774 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:39.062777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:39.062804 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:39.065334 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:39.075956 (+ 10622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:39.075962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:39.075965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:39.076153 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:39.076728 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:39.076738 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:39.076741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:39.077013 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:39.077185 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:39.077940 (+ 755us) server_negotiation.cc:299] Negotiation successful
0927 06:54:39.078229 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":184,"threads_started":1}
I20260927 06:54:39.079561 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:39.051797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46301 (local address 127.0.97.3:59301)
0927 06:54:39.052577 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:39.052617 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:39.054212 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:39.054846 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:39.054854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:39.055304 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:39.055889 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:39.055905 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:39.056881 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:39.056884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:39.058720 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:39.058736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:39.059667 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:39.059674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:39.059886 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:39.060535 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:39.060561 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:39.062574 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:39.065472 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:39.065481 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:39.065496 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:39.065897 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:39.076318 (+ 10421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:39.076324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:39.076328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:39.076586 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:39.077159 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:39.077169 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:39.077563 (+ 394us) client_negotiation.cc:769] Sending connection context
0927 06:54:39.077840 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 06:54:39.078432 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":293,"threads_started":1}
I20260927 06:54:39.081174 3941 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46301
I20260927 06:54:39.081709 3941 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:39.082677 3941 heartbeater.cc:511] Master 127.0.97.62:46301 requested a full tablet report, sending...
I20260927 06:54:39.084231 3484 ts_manager.cc:194] Registered new tserver with Master: 01e3ad4b355149df82ed17438a478a68 (127.0.97.3:35353)
I20260927 06:54:39.085041 3484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59301
I20260927 06:54:39.090230 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 06:54:39 dist-test-slave-f892 krb5kdc[3439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492079, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 06:54:39 dist-test-slave-f892 krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492079, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:39.136658 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:39.122971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.0.97.62:46301)
0927 06:54:39.123223 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:39.123232 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:39.123565 (+ 333us) server_negotiation.cc:407] Connection header received
0927 06:54:39.123781 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:39.123789 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:39.123868 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:39.123976 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:39.125276 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:39.126069 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:39.127270 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:39.127698 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:39.127839 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:39.131500 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:39.131505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:39.131508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:39.131541 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:39.134216 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:39.134794 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:39.134800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:39.134804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:39.134925 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:39.135314 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:39.135319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:39.135323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:39.135627 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:39.135862 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:39.136134 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 06:54:39.136349 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260927 06:54:39.137200 3484 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35416
I20260927 06:54:39.138993 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 3545
Sep 27 06:54:39 dist-test-slave-f892 krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492079, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:54:39.154475 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:39.139994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53544 (local address 127.0.97.1:43185)
0927 06:54:39.140566 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:39.140573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:39.140594 (+ 21us) server_negotiation.cc:407] Connection header received
0927 06:54:39.140643 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:39.140649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:39.140837 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:39.140991 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:39.142224 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:39.143173 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:39.144455 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:39.145079 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:39.145218 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:39.148920 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:39.148931 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:39.148946 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:39.148984 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:39.151831 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:39.152525 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:39.152532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:39.152534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:39.152646 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:39.153004 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:39.153010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:39.153014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:39.153338 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:39.153641 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:39.153876 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 06:54:39.154148 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":282,"threads_started":1}
W20260927 06:54:39.155777 3668 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53544
W20260927 06:54:39.156804 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 3545: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:54:39.157001 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 3545
I20260927 06:54:39.174120 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 3679
Sep 27 06:54:39 dist-test-slave-f892 krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492079, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:54:39.190157 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:39.175271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51364 (local address 127.0.97.2:40029)
0927 06:54:39.175817 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:39.175824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:39.175850 (+ 26us) server_negotiation.cc:407] Connection header received
0927 06:54:39.175919 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:39.175925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:39.176131 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:39.176272 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:39.177426 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:39.178602 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:39.179826 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:39.180352 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:39.180451 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:39.184517 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:39.184527 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:39.184541 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:39.184588 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:39.187355 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:39.188113 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:39.188120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:39.188123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:39.188259 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:39.188636 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:39.188639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:39.188641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:39.188987 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:39.189203 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:39.189560 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 06:54:39.189780 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":313,"threads_started":1}
W20260927 06:54:39.191354 3802 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51364
W20260927 06:54:39.192438 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 3679: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:54:39.192619 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 3679
I20260927 06:54:39.207898 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 3813
Sep 27 06:54:39 dist-test-slave-f892 krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492079, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 06:54:39.230434 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:39.209796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42542 (local address 127.0.97.3:35353)
0927 06:54:39.211540 (+ 1744us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:39.211548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:39.211583 (+ 35us) server_negotiation.cc:407] Connection header received
0927 06:54:39.211866 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:39.211876 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:39.212096 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:39.212243 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:39.213736 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:39.215538 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:39.216875 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:39.217481 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:39.217595 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:39.221768 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:39.221779 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:39.221792 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:39.221829 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:39.225752 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:39.227366 (+ 1614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:39.227404 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:39.227411 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:39.227686 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:39.228552 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:39.228558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:39.228562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:39.229051 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:39.229415 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:39.229727 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 06:54:39.230052 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1493,"thread_start_us":1351,"threads_started":1}
W20260927 06:54:39.231739 3936 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42542
W20260927 06:54:39.232892 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 3813: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:54:39.233086 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 3813
I20260927 06:54:39.248157 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 3452
I20260927 06:54:39.249238 3514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:39.364226 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 3452
2026-09-27T06:54:39Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3765 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:54:39 dist-test-slave-f892 krb5kdc[3959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:54:39 dist-test-slave-f892 krb5kdc[3959](info): set up 2 sockets
Sep 27 06:54:39 dist-test-slave-f892 krb5kdc[3959](info): commencing operation
krb5kdc: starting...
W20260927 06:54:40.534251 388 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:54:40 dist-test-slave-f892 krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:54:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:54:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:54:40.736804 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37753
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:41.086161 3972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:41.086440 3972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:41.086602 3972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:41.093600 3972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:54:41.093693 3972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:41.093779 3972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:41.093881 3972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:54:41.093941 3972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:54:41.103082 3972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:41.104333 3972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:41.105609 3972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:41.117089 3980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:41.117152 3978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:41.117417 3977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:41.118119 3972 server_base.cc:1034] running on GCE node
I20260927 06:54:41.118902 3972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:41.120163 3972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:41.121390 3972 hybrid_clock.cc:648] HybridClock initialized: now 1790492081121357 us; error 48 us; skew 500 ppm
I20260927 06:54:41.121721 3972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:41 dist-test-slave-f892 krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:41.124739 3972 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:54:41.126169 3972 webserver.cc:460] Webserver started at http://127.0.97.62:39615/ using document root <none> and password file <none>
I20260927 06:54:41.126799 3972 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:41.126937 3972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:41.127230 3972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:41.129914 3972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "542c9bba21074ec383a7ba407ef25ac7"
format_stamp: "Formatted at 2026-09-27 06:54:41 on dist-test-slave-f892"
server_key: "66ed8f8272fa205f15673cf7de280b7f"
server_key_iv: "5d03447bc55d123ff061a5b5ea4fb019"
server_key_version: "encryptionkey@0"
I20260927 06:54:41.130702 3972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "542c9bba21074ec383a7ba407ef25ac7"
format_stamp: "Formatted at 2026-09-27 06:54:41 on dist-test-slave-f892"
server_key: "66ed8f8272fa205f15673cf7de280b7f"
server_key_iv: "5d03447bc55d123ff061a5b5ea4fb019"
server_key_version: "encryptionkey@0"
I20260927 06:54:41.135300 3972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 06:54:41.138343 3987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:41.139482 3972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 06:54:41.139648 3972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "542c9bba21074ec383a7ba407ef25ac7"
format_stamp: "Formatted at 2026-09-27 06:54:41 on dist-test-slave-f892"
server_key: "66ed8f8272fa205f15673cf7de280b7f"
server_key_iv: "5d03447bc55d123ff061a5b5ea4fb019"
server_key_version: "encryptionkey@0"
I20260927 06:54:41.139819 3972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:41.158874 3972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:41.163174 3972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:41.163441 3972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:41.185259 3972 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37753
I20260927 06:54:41.185277 4038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37753 every 8 connection(s)
I20260927 06:54:41.189764 3972 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:54:41.194027 4039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:41.195508 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 3972
I20260927 06:54:41.195724 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:54:41.196484 388 external_mini_cluster.cc:1428] Setting key 4cc7a5a858d00a753f4d16ddf4022155
I20260927 06:54:41.206732 4039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7: Bootstrap starting.
Sep 27 06:54:41 dist-test-slave-f892 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:41.209973 4039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:41.211047 4039 log.cc:824] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:41.213694 4039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7: No bootstrap required, opened a new log
I20260927 06:54:41.217221 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:41.198506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.0.97.62:37753)
0927 06:54:41.199225 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:41.199240 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:41.199300 (+ 60us) server_negotiation.cc:407] Connection header received
0927 06:54:41.200815 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:41.200839 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:41.201215 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:41.201593 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:41.202957 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:41.204227 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:41.205412 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:41.206134 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:41.206322 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:41.210589 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:41.210596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:41.210609 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:41.210645 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:41.213528 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:41.214265 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:41.214271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:41.214277 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:41.214422 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:41.214786 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:41.214794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:41.214796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:41.215259 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:41.215576 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:41.215942 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 06:54:41.216331 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":323,"threads_started":1}
I20260927 06:54:41.221635 4039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542c9bba21074ec383a7ba407ef25ac7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37753 } }
I20260927 06:54:41.221966 4039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:41.222103 4039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 542c9bba21074ec383a7ba407ef25ac7, State: Initialized, Role: FOLLOWER
I20260927 06:54:41.222582 4039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542c9bba21074ec383a7ba407ef25ac7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37753 } }
I20260927 06:54:41.222777 4039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:54:41.222879 4039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:54:41.223012 4039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:41.224972 4039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542c9bba21074ec383a7ba407ef25ac7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37753 } }
I20260927 06:54:41.225384 4039 leader_election.cc:302] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 542c9bba21074ec383a7ba407ef25ac7; no voters:
I20260927 06:54:41.225987 4039 leader_election.cc:288] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:54:41.226166 4044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:41.226842 4044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7 [term 1 LEADER]: Becoming Leader. State: Replica: 542c9bba21074ec383a7ba407ef25ac7, State: Running, Role: LEADER
I20260927 06:54:41.227388 4044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542c9bba21074ec383a7ba407ef25ac7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37753 } }
I20260927 06:54:41.228117 4039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:54:41.230278 4045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "542c9bba21074ec383a7ba407ef25ac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542c9bba21074ec383a7ba407ef25ac7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37753 } } }
I20260927 06:54:41.230726 4045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:41.231280 4046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 542c9bba21074ec383a7ba407ef25ac7. Latest consensus state: current_term: 1 leader_uuid: "542c9bba21074ec383a7ba407ef25ac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542c9bba21074ec383a7ba407ef25ac7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37753 } } }
I20260927 06:54:41.232180 4046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:41.245334 4052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:54:41.250808 4052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:54:41.259161 4052 catalog_manager.cc:1347] Generated new cluster ID: efb60fbdbd4242949890c4da38fce6fb
I20260927 06:54:41.259276 4052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:54:41.284409 4052 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:54:41.285753 4052 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:54:41.299190 4052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 542c9bba21074ec383a7ba407ef25ac7: Generated new TSK 0
I20260927 06:54:41.300060 4052 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 06:54:41.345206 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37753
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:41.678113 4065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:41.678392 4065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:41.678483 4065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:41.685647 4065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:41.685755 4065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:41.685864 4065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:54:41.695374 4065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:41.696583 4065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:41.697810 4065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:41.710749 4070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:41.710774 4073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:41.710798 4071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:41.712109 4065 server_base.cc:1034] running on GCE node
I20260927 06:54:41.712780 4065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:41.713758 4065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:41.714979 4065 hybrid_clock.cc:648] HybridClock initialized: now 1790492081714904 us; error 70 us; skew 500 ppm
I20260927 06:54:41.715305 4065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:41 dist-test-slave-f892 krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:41.718432 4065 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:54:41.719743 4065 webserver.cc:460] Webserver started at http://127.0.97.1:35335/ using document root <none> and password file <none>
I20260927 06:54:41.720359 4065 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:41.720454 4065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:41.720695 4065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:41.723956 4065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "8304aefe71dd49cd8be38c63178838b5"
format_stamp: "Formatted at 2026-09-27 06:54:41 on dist-test-slave-f892"
server_key: "4e510c53dd6fd324058a8f182bd9a068"
server_key_iv: "f93b7d29f44fe92bedb89d635832954b"
server_key_version: "encryptionkey@0"
I20260927 06:54:41.724687 4065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8304aefe71dd49cd8be38c63178838b5"
format_stamp: "Formatted at 2026-09-27 06:54:41 on dist-test-slave-f892"
server_key: "4e510c53dd6fd324058a8f182bd9a068"
server_key_iv: "f93b7d29f44fe92bedb89d635832954b"
server_key_version: "encryptionkey@0"
I20260927 06:54:41.729673 4065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 06:54:41.732997 4080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:41.734372 4065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 06:54:41.734498 4065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "8304aefe71dd49cd8be38c63178838b5"
format_stamp: "Formatted at 2026-09-27 06:54:41 on dist-test-slave-f892"
server_key: "4e510c53dd6fd324058a8f182bd9a068"
server_key_iv: "f93b7d29f44fe92bedb89d635832954b"
server_key_version: "encryptionkey@0"
I20260927 06:54:41.734652 4065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:41.748055 4065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:41.752411 4065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:41.752650 4065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:41.753590 4065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:41.754895 4065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:41.754964 4065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:41.755061 4065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:41.755100 4065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:41.787086 4065 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45667
I20260927 06:54:41.787130 4192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45667 every 8 connection(s)
I20260927 06:54:41.788354 4065 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:54:41.794242 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 4065
I20260927 06:54:41.794445 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:54:41.795224 388 external_mini_cluster.cc:1428] Setting key 647b2679f745f90e2fa0a53201f38a42
Sep 27 06:54:41 dist-test-slave-f892 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:41.811683 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:41.791214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57947 (local address 127.0.97.62:37753)
0927 06:54:41.791690 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:41.791695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:41.792387 (+ 692us) server_negotiation.cc:407] Connection header received
0927 06:54:41.794173 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:41.794179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:41.794268 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:41.794389 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:41.796426 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:41.797461 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:41.800000 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:41.800392 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:41.800485 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:41.804808 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:41.804818 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:41.804826 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:41.804878 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:41.808357 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:41.809094 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:41.809102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:41.809109 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:41.809249 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:41.809674 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:41.809682 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:41.809688 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:41.810098 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:41.810248 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:41.810961 (+ 713us) server_negotiation.cc:299] Negotiation successful
0927 06:54:41.811207 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":269,"threads_started":1}
I20260927 06:54:41.812384 4196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:41.791536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37753 (local address 127.0.97.1:57947)
0927 06:54:41.792214 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:41.792252 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:41.793876 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:41.794698 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:41.794712 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:41.795455 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:41.796246 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:41.796271 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:41.797614 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:41.797620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:41.799852 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:41.799869 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:41.800617 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:41.800626 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:41.800936 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:41.801642 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:41.801674 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:41.804574 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:41.808507 (+ 3933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:41.808515 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:41.808528 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:41.808962 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:41.809379 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:41.809382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:41.809384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:41.809549 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:41.810234 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:41.810240 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:41.810586 (+ 346us) client_negotiation.cc:769] Sending connection context
0927 06:54:41.810808 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 06:54:41.811172 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":275,"threads_started":1}
I20260927 06:54:41.813890 4193 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37753
I20260927 06:54:41.814230 4193 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:41.814924 4193 heartbeater.cc:511] Master 127.0.97.62:37753 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 06:54:41.817011 4004 ts_manager.cc:194] Registered new tserver with Master: 8304aefe71dd49cd8be38c63178838b5 (127.0.97.1:45667)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 06:54:41.818856 4004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 06:54:41.839645 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37753
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:42.151271 4199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:42.151590 4199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:42.151750 4199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:42.158689 4199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:42.158826 4199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:42.159005 4199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:54:42.168309 4199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:42.169562 4199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:42.170787 4199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:42.182971 4204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:42.182919 4205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:42.182940 4207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:42.184029 4199 server_base.cc:1034] running on GCE node
I20260927 06:54:42.184630 4199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:42.185529 4199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:42.186754 4199 hybrid_clock.cc:648] HybridClock initialized: now 1790492082186696 us; error 50 us; skew 500 ppm
I20260927 06:54:42.187045 4199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:42 dist-test-slave-f892 krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:42.190100 4199 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:54:42.191444 4199 webserver.cc:460] Webserver started at http://127.0.97.2:43879/ using document root <none> and password file <none>
I20260927 06:54:42.192024 4199 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:42.192119 4199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:42.192375 4199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:42.195083 4199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "e4218bcc644e4864ac01e5d98530cc44"
format_stamp: "Formatted at 2026-09-27 06:54:42 on dist-test-slave-f892"
server_key: "857eb3fb13e3fc1255f02780f1107312"
server_key_iv: "1306f39004aeac705e6f17545534bce8"
server_key_version: "encryptionkey@0"
I20260927 06:54:42.195820 4199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e4218bcc644e4864ac01e5d98530cc44"
format_stamp: "Formatted at 2026-09-27 06:54:42 on dist-test-slave-f892"
server_key: "857eb3fb13e3fc1255f02780f1107312"
server_key_iv: "1306f39004aeac705e6f17545534bce8"
server_key_version: "encryptionkey@0"
I20260927 06:54:42.200577 4199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 06:54:42.203817 4214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:42.204993 4199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 06:54:42.205190 4199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "e4218bcc644e4864ac01e5d98530cc44"
format_stamp: "Formatted at 2026-09-27 06:54:42 on dist-test-slave-f892"
server_key: "857eb3fb13e3fc1255f02780f1107312"
server_key_iv: "1306f39004aeac705e6f17545534bce8"
server_key_version: "encryptionkey@0"
I20260927 06:54:42.205371 4199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:42.241346 4199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:42.245429 4199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:42.245708 4199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:42.246734 4199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:42.248011 4199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:42.248090 4199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:42.248265 4199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:42.248314 4199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:42.280857 4199 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41339
I20260927 06:54:42.280912 4326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41339 every 8 connection(s)
I20260927 06:54:42.282191 4199 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:54:42.286863 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 4199
I20260927 06:54:42.287074 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:54:42.287474 388 external_mini_cluster.cc:1428] Setting key af5499d139c9d6387fda0daadb3a5938
Sep 27 06:54:42 dist-test-slave-f892 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:42.302196 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:42.284631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45235 (local address 127.0.97.62:37753)
0927 06:54:42.284797 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:42.284801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:42.285797 (+ 996us) server_negotiation.cc:407] Connection header received
0927 06:54:42.287684 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:42.287690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:42.287774 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:42.287882 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:42.289723 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:42.290871 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:42.292344 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:42.293044 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:42.293200 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:42.296624 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:42.296629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:42.296632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:42.296662 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:42.299242 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:42.299970 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:42.299976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:42.299980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:42.300113 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:42.300528 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:42.300534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:42.300538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:42.300794 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:42.300945 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:42.301612 (+ 667us) server_negotiation.cc:299] Negotiation successful
0927 06:54:42.301824 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 06:54:42.303092 4329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:42.284977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37753 (local address 127.0.97.2:45235)
0927 06:54:42.285648 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:42.285686 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:42.287418 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:42.288122 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:42.288133 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:42.288801 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:42.289561 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:42.289580 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:42.291014 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:42.291018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:42.292184 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:42.292194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:42.293249 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:42.293253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:42.293429 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:42.294128 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:42.294153 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:42.296413 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:42.299389 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:42.299398 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:42.299414 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:42.299841 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:42.300226 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:42.300231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:42.300235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:42.300417 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:42.300924 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:42.300931 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:42.301259 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 06:54:42.301520 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 06:54:42.302063 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":287,"threads_started":1}
I20260927 06:54:42.304632 4327 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37753
I20260927 06:54:42.304922 4327 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 06:54:42.305672 4327 heartbeater.cc:511] Master 127.0.97.62:37753 requested a full tablet report, sending...
I20260927 06:54:42.307183 4004 ts_manager.cc:194] Registered new tserver with Master: e4218bcc644e4864ac01e5d98530cc44 (127.0.97.2:41339)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 06:54:42.308087 4004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 06:54:42.331616 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37753
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:42.671233 4332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:42.671531 4332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:42.671636 4332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:42.678834 4332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:42.678937 4332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:42.679052 4332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:54:42.688396 4332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:42.689705 4332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:42.691148 4332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:42.703588 4337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:42.703617 4338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:42.703830 4340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:42.704684 4332 server_base.cc:1034] running on GCE node
I20260927 06:54:42.705222 4332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:42.706156 4332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:42.707399 4332 hybrid_clock.cc:648] HybridClock initialized: now 1790492082707343 us; error 50 us; skew 500 ppm
I20260927 06:54:42.707731 4332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:42 dist-test-slave-f892 krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:42.711198 4332 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:54:42.712707 4332 webserver.cc:460] Webserver started at http://127.0.97.3:39311/ using document root <none> and password file <none>
I20260927 06:54:42.713407 4332 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:42.713558 4332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:42.713858 4332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:42.716778 4332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "b6dcd23c0c2145898d9cdc6654aab500"
format_stamp: "Formatted at 2026-09-27 06:54:42 on dist-test-slave-f892"
server_key: "b14ef23e83772141b149381d1095e394"
server_key_iv: "46d025b67a4528c3fa02da5a3060cd1d"
server_key_version: "encryptionkey@0"
I20260927 06:54:42.717572 4332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b6dcd23c0c2145898d9cdc6654aab500"
format_stamp: "Formatted at 2026-09-27 06:54:42 on dist-test-slave-f892"
server_key: "b14ef23e83772141b149381d1095e394"
server_key_iv: "46d025b67a4528c3fa02da5a3060cd1d"
server_key_version: "encryptionkey@0"
I20260927 06:54:42.722604 4332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 06:54:42.725852 4347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:42.727205 4332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:54:42.727377 4332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "b6dcd23c0c2145898d9cdc6654aab500"
format_stamp: "Formatted at 2026-09-27 06:54:42 on dist-test-slave-f892"
server_key: "b14ef23e83772141b149381d1095e394"
server_key_iv: "46d025b67a4528c3fa02da5a3060cd1d"
server_key_version: "encryptionkey@0"
I20260927 06:54:42.727567 4332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:42.757332 4332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:42.761436 4332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:42.761724 4332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:42.762667 4332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:42.763957 4332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:42.764061 4332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:42.764213 4332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:42.764295 4332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:42.796525 4332 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36423
I20260927 06:54:42.796567 4459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36423 every 8 connection(s)
I20260927 06:54:42.797719 4332 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:54:42.800714 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 4332
I20260927 06:54:42.800886 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:54:42.801467 388 external_mini_cluster.cc:1428] Setting key 9b64d814a95d0b6b9b6312373abfc9be
Sep 27 06:54:42 dist-test-slave-f892 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:42.817834 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:42.800097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40147 (local address 127.0.97.62:37753)
0927 06:54:42.800301 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:42.800307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:42.802077 (+ 1770us) server_negotiation.cc:407] Connection header received
0927 06:54:42.804486 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:42.804491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:42.804584 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:42.804688 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:42.806135 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:42.806938 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:42.808448 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:42.808838 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:42.808972 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:42.812269 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:42.812273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:42.812277 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:42.812305 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:42.814887 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:42.815764 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:42.815767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:42.815769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:42.815936 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:42.816404 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:42.816409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:42.816413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:42.816755 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:42.816862 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:42.817451 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 06:54:42.817606 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 06:54:42.818754 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:42.800693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37753 (local address 127.0.97.3:40147)
0927 06:54:42.801797 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:42.801882 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:42.804209 (+ 2327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:42.804911 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:42.804919 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:42.805390 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:42.805913 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:42.805928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:42.807079 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:42.807085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:42.808280 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:42.808291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:42.809058 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:42.809065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:42.809319 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:42.809979 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:42.810043 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:42.812069 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:42.815035 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:42.815045 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:42.815060 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:42.815638 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:42.816059 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:42.816064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:42.816068 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:42.816280 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:42.816864 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:42.816869 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:42.817155 (+ 286us) client_negotiation.cc:769] Sending connection context
0927 06:54:42.817360 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 06:54:42.817705 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":455,"threads_started":1}
I20260927 06:54:42.819965 4460 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37753
I20260927 06:54:42.820268 4460 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:42.820948 4460 heartbeater.cc:511] Master 127.0.97.62:37753 requested a full tablet report, sending...
I20260927 06:54:42.821696 4193 heartbeater.cc:503] Master 127.0.97.62:37753 was elected leader, sending a full tablet report...
I20260927 06:54:42.822403 4004 ts_manager.cc:194] Registered new tserver with Master: b6dcd23c0c2145898d9cdc6654aab500 (127.0.97.3:36423)
I20260927 06:54:42.823294 4004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40147
I20260927 06:54:42.828673 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 06:54:42 dist-test-slave-f892 krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 06:54:42 dist-test-slave-f892 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:54:42.873554 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:42.859643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.0.97.1:45667)
0927 06:54:42.860244 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:42.860251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:42.860276 (+ 25us) server_negotiation.cc:407] Connection header received
0927 06:54:42.860333 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:42.860339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:42.860537 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:42.860658 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:42.861856 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:42.862876 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:42.864042 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:42.864594 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:42.864685 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:42.868372 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:42.868383 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:42.868397 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:42.868433 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:42.871386 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:42.871916 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:42.871922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:42.871925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:42.872056 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:42.872349 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:42.872352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:42.872354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:42.872676 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:42.872883 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:42.873088 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 06:54:42.873271 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":321,"threads_started":1}
Sep 27 06:54:42 dist-test-slave-f892 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:42.900108 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:42.880837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57288 (local address 127.0.97.62:37753)
0927 06:54:42.881030 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:42.881036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:42.881177 (+ 141us) server_negotiation.cc:407] Connection header received
0927 06:54:42.881352 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:42.881358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:42.881450 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:42.881568 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:42.883548 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:42.884676 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:42.886793 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:42.887648 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:42.887794 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:42.893065 (+ 5271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:42.893073 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:42.893078 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:42.893117 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:42.896463 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:42.897212 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:42.897220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:42.897226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:42.897374 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:42.897801 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:42.897809 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:42.897815 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:42.898270 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:42.898517 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:42.899519 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0927 06:54:42.899753 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 06:54:42.900734 4004 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57288
I20260927 06:54:42.901836 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 4065
I20260927 06:54:42.918320 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:42.903080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.0.97.1:45667)
0927 06:54:42.903276 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:42.903282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:42.903301 (+ 19us) server_negotiation.cc:407] Connection header received
0927 06:54:42.903358 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:42.903366 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:42.903461 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:42.903587 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:42.904933 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:42.906156 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:42.907551 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:42.907962 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:42.908070 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:42.910334 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:42.910338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:42.910342 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:42.910372 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:42.916165 (+ 5793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:42.916730 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:42.916734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:42.916736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:42.916848 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:42.917135 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:42.917137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:42.917139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:42.917374 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:42.917545 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:42.917784 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 06:54:42.917937 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 06:54:42.918877 4188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:42.998795 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 4065
I20260927 06:54:43.015065 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 4199
Sep 27 06:54:43 dist-test-slave-f892 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:54:43.030722 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:43.015930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.0.97.2:41339)
0927 06:54:43.016419 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:43.016426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:43.016449 (+ 23us) server_negotiation.cc:407] Connection header received
0927 06:54:43.016506 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:43.016512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:43.016712 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:43.016836 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:43.017961 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:43.019024 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:43.020341 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:43.020856 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:43.020949 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:43.024999 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:43.025012 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:43.025026 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:43.025064 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:43.027971 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:43.028654 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:43.028661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:43.028663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:43.028800 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:43.029149 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:43.029152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:43.029154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:43.029590 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:43.029797 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:43.030134 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 06:54:43.030332 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":252,"threads_started":1}
I20260927 06:54:43.032095 4322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:43.114226 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 4199
I20260927 06:54:43.130158 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 4332
Sep 27 06:54:43 dist-test-slave-f892 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 06:54:43.145684 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:43.130984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40634 (local address 127.0.97.3:36423)
0927 06:54:43.131616 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:43.131623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:43.131650 (+ 27us) server_negotiation.cc:407] Connection header received
0927 06:54:43.131718 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:43.131723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:43.131968 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:43.132130 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:43.133510 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:43.134623 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:43.135851 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:43.136383 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:43.136516 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:43.140453 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:43.140463 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:43.140476 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:43.140506 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:43.143127 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:43.143752 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:43.143762 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:43.143770 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:43.143884 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:43.144234 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:43.144240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:43.144243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:43.144570 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:43.144905 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:43.145107 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 06:54:43.145353 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":335,"threads_started":1}
I20260927 06:54:43.146917 4455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:43.234771 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 4332
I20260927 06:54:43.251348 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 3972
I20260927 06:54:43.252434 4034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:43.352368 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 3972
2026-09-27T06:54:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3985 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:54:43 dist-test-slave-f892 krb5kdc[4479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:54:43 dist-test-slave-f892 krb5kdc[4479](info): set up 2 sockets
Sep 27 06:54:43 dist-test-slave-f892 krb5kdc[4479](info): commencing operation
krb5kdc: starting...
W20260927 06:54:44.442397 388 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:54:44 dist-test-slave-f892 krb5kdc[4479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:54:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:54:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:54:44.627692 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43517
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:44.936177 4492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:44.936443 4492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:44.936642 4492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:44.943588 4492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:54:44.943703 4492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:44.943786 4492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:44.943884 4492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:54:44.943971 4492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:54:44.953429 4492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:44.954727 4492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:44.956030 4492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:44.967056 4497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:44.967020 4500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:44.967152 4498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:44.968240 4492 server_base.cc:1034] running on GCE node
I20260927 06:54:44.968885 4492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:44.970151 4492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:44.971372 4492 hybrid_clock.cc:648] HybridClock initialized: now 1790492084971325 us; error 27 us; skew 500 ppm
I20260927 06:54:44.971704 4492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:44 dist-test-slave-f892 krb5kdc[4479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:44.974587 4492 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:54:44.975710 4492 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:44.975831 4492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:44.976141 4492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:44.978965 4492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "a52da375a2f44d64a4d3295285aa6625"
format_stamp: "Formatted at 2026-09-27 06:54:44 on dist-test-slave-f892"
server_key: "dbbc47d947f2e5d1d00d55226fb1bb83"
server_key_iv: "bb8b77f08b39f4187c2b30c7a19b7577"
server_key_version: "encryptionkey@0"
I20260927 06:54:44.979709 4492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "a52da375a2f44d64a4d3295285aa6625"
format_stamp: "Formatted at 2026-09-27 06:54:44 on dist-test-slave-f892"
server_key: "dbbc47d947f2e5d1d00d55226fb1bb83"
server_key_iv: "bb8b77f08b39f4187c2b30c7a19b7577"
server_key_version: "encryptionkey@0"
I20260927 06:54:44.984295 4492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 06:54:44.987521 4506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:44.988783 4492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 06:54:44.988967 4492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "a52da375a2f44d64a4d3295285aa6625"
format_stamp: "Formatted at 2026-09-27 06:54:44 on dist-test-slave-f892"
server_key: "dbbc47d947f2e5d1d00d55226fb1bb83"
server_key_iv: "bb8b77f08b39f4187c2b30c7a19b7577"
server_key_version: "encryptionkey@0"
I20260927 06:54:44.989185 4492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:45.018283 4492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:45.022082 4492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:45.022342 4492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:45.040611 4492 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43517
I20260927 06:54:45.040630 4557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43517 every 8 connection(s)
I20260927 06:54:45.041498 4492 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:54:45.044265 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 4492
I20260927 06:54:45.044481 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:54:45.045254 388 external_mini_cluster.cc:1428] Setting key f1966df36dd8cffbfa277f08459b91a9
I20260927 06:54:45.045598 4558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:45.059747 4558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625: Bootstrap starting.
Sep 27 06:54:45 dist-test-slave-f892 krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:45.063509 4558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:45.064787 4558 log.cc:824] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:45.068020 4558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625: No bootstrap required, opened a new log
I20260927 06:54:45.069363 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:45.047421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.0.97.62:43517)
0927 06:54:45.048169 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:45.048179 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:45.048220 (+ 41us) server_negotiation.cc:407] Connection header received
0927 06:54:45.050194 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:45.050214 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:45.050571 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:45.050913 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:45.052636 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:45.053895 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:45.055769 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:45.056516 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:45.056642 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:45.061186 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:45.061201 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:45.061221 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:45.061273 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:45.065058 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:45.065827 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:45.065834 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:45.065839 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:45.065995 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:45.066424 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:45.066429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:45.066432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:45.067080 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:45.067369 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:45.067622 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 06:54:45.068035 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":361,"threads_started":1}
I20260927 06:54:45.076484 4558 raft_consensus.cc:348] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52da375a2f44d64a4d3295285aa6625" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43517 } }
I20260927 06:54:45.076794 4558 raft_consensus.cc:374] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:45.076876 4558 raft_consensus.cc:729] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a52da375a2f44d64a4d3295285aa6625, State: Initialized, Role: FOLLOWER
I20260927 06:54:45.077330 4558 consensus_queue.cc:260] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52da375a2f44d64a4d3295285aa6625" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43517 } }
I20260927 06:54:45.077521 4558 raft_consensus.cc:388] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:54:45.077662 4558 raft_consensus.cc:482] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:54:45.077778 4558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:45.080277 4558 raft_consensus.cc:504] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52da375a2f44d64a4d3295285aa6625" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43517 } }
I20260927 06:54:45.080693 4558 leader_election.cc:302] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a52da375a2f44d64a4d3295285aa6625; no voters:
I20260927 06:54:45.081184 4558 leader_election.cc:288] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:54:45.081328 4563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:45.082134 4563 raft_consensus.cc:686] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625 [term 1 LEADER]: Becoming Leader. State: Replica: a52da375a2f44d64a4d3295285aa6625, State: Running, Role: LEADER
I20260927 06:54:45.082633 4563 consensus_queue.cc:237] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52da375a2f44d64a4d3295285aa6625" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43517 } }
I20260927 06:54:45.083566 4558 sys_catalog.cc:564] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:54:45.085251 4565 sys_catalog.cc:455] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a52da375a2f44d64a4d3295285aa6625. Latest consensus state: current_term: 1 leader_uuid: "a52da375a2f44d64a4d3295285aa6625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52da375a2f44d64a4d3295285aa6625" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43517 } } }
I20260927 06:54:45.085557 4565 sys_catalog.cc:458] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:45.086262 4564 sys_catalog.cc:455] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a52da375a2f44d64a4d3295285aa6625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52da375a2f44d64a4d3295285aa6625" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43517 } } }
I20260927 06:54:45.086527 4564 sys_catalog.cc:458] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:45.087098 4569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:54:45.091830 4569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:54:45.100826 4569 catalog_manager.cc:1347] Generated new cluster ID: 6a64b334db8a4e72acd67044f029bdc6
I20260927 06:54:45.100934 4569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:54:45.121677 4569 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:54:45.123044 4569 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:54:45.139777 4569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625: Generated new TSK 0
I20260927 06:54:45.140792 4569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 06:54:45.190723 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43517
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:45.504814 4584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:45.505111 4584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:45.505278 4584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:45.512564 4584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:45.512719 4584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:45.512905 4584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:54:45.522459 4584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:45.523707 4584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:45.524956 4584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:45.537321 4592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:45.537379 4589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:45.537353 4590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:45.538303 4584 server_base.cc:1034] running on GCE node
I20260927 06:54:45.538884 4584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:45.539654 4584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:45.540870 4584 hybrid_clock.cc:648] HybridClock initialized: now 1790492085540824 us; error 37 us; skew 500 ppm
I20260927 06:54:45.541163 4584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:45 dist-test-slave-f892 krb5kdc[4479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:45.544068 4584 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:54:45.545058 4584 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:45.545184 4584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:45.545475 4584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:45.548399 4584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "7e24ec310e404d0a9f5dbde7b5f3f10c"
format_stamp: "Formatted at 2026-09-27 06:54:45 on dist-test-slave-f892"
server_key: "027b69f43ba55ab9abd32448729e74d0"
server_key_iv: "090e3b3d74c3721bb09f021fba2b6182"
server_key_version: "encryptionkey@0"
I20260927 06:54:45.549223 4584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7e24ec310e404d0a9f5dbde7b5f3f10c"
format_stamp: "Formatted at 2026-09-27 06:54:45 on dist-test-slave-f892"
server_key: "027b69f43ba55ab9abd32448729e74d0"
server_key_iv: "090e3b3d74c3721bb09f021fba2b6182"
server_key_version: "encryptionkey@0"
I20260927 06:54:45.554176 4584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260927 06:54:45.557430 4598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:45.558739 4584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:54:45.558911 4584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "7e24ec310e404d0a9f5dbde7b5f3f10c"
format_stamp: "Formatted at 2026-09-27 06:54:45 on dist-test-slave-f892"
server_key: "027b69f43ba55ab9abd32448729e74d0"
server_key_iv: "090e3b3d74c3721bb09f021fba2b6182"
server_key_version: "encryptionkey@0"
I20260927 06:54:45.559122 4584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:45.574496 4584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:45.578404 4584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:45.578663 4584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:45.579496 4584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:45.580761 4584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:45.580881 4584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:45.581012 4584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:45.581099 4584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:45.613293 4584 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35595
I20260927 06:54:45.613338 4710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35595 every 8 connection(s)
I20260927 06:54:45.614130 4584 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:54:45.620042 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 4584
I20260927 06:54:45.620252 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:54:45.620879 388 external_mini_cluster.cc:1428] Setting key 285143de118f709381f90e6258b45efa
Sep 27 06:54:45 dist-test-slave-f892 krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:45.633422 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:45.616415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54851 (local address 127.0.97.62:43517)
0927 06:54:45.616799 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:45.616804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:45.617720 (+ 916us) server_negotiation.cc:407] Connection header received
0927 06:54:45.619413 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:45.619417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:45.619479 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:45.619574 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:45.621059 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:45.621911 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:45.623699 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:45.624462 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:45.624612 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:45.627793 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:45.627797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:45.627801 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:45.627830 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:45.630314 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:45.631194 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:45.631198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:45.631200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:45.631309 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:45.631771 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:45.631774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:45.631776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:45.632041 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:45.632150 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:45.632995 (+ 845us) server_negotiation.cc:299] Negotiation successful
0927 06:54:45.633159 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":195,"threads_started":1}
I20260927 06:54:45.634897 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:45.616946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43517 (local address 127.0.97.1:54851)
0927 06:54:45.617571 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:45.617605 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:45.619191 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:45.619751 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:45.619758 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:45.620260 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:45.620837 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:45.620853 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:45.622073 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:45.622078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:45.623555 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:45.623567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:45.624651 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:45.624655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:45.624829 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:45.625489 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:45.625514 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:45.627576 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:45.630440 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:45.630451 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:45.630478 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:45.631080 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:45.631409 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:45.631416 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:45.631423 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:45.631649 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:45.632168 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:45.632179 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:45.632623 (+ 444us) client_negotiation.cc:769] Sending connection context
0927 06:54:45.632910 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 06:54:45.633457 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":250,"threads_started":1}
I20260927 06:54:45.636351 4711 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43517
I20260927 06:54:45.636691 4711 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:45.637517 4711 heartbeater.cc:511] Master 127.0.97.62:43517 requested a full tablet report, sending...
I20260927 06:54:45.639796 4523 ts_manager.cc:194] Registered new tserver with Master: 7e24ec310e404d0a9f5dbde7b5f3f10c (127.0.97.1:35595)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 06:54:45.641333 4523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 06:54:45.663076 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43517
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:45.968324 4717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:45.968554 4717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:45.968724 4717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:45.975620 4717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:45.975734 4717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:45.975912 4717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:54:45.985020 4717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:45.986351 4717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:45.987654 4717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:46.000627 4722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:46.000661 4725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:46.000597 4723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:46.001314 4717 server_base.cc:1034] running on GCE node
I20260927 06:54:46.001889 4717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:46.002794 4717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:46.004015 4717 hybrid_clock.cc:648] HybridClock initialized: now 1790492086003963 us; error 57 us; skew 500 ppm
I20260927 06:54:46.004316 4717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:46 dist-test-slave-f892 krb5kdc[4479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:46.007331 4717 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:54:46.008400 4717 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:46.008523 4717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:46.008834 4717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:46.011608 4717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "cfd1b9a59e8b42b8ac6673daff3b190d"
format_stamp: "Formatted at 2026-09-27 06:54:46 on dist-test-slave-f892"
server_key: "a63385c84414a31bc9d83b461789a7ef"
server_key_iv: "e7ae2d664928708d74b3be877bd138c3"
server_key_version: "encryptionkey@0"
I20260927 06:54:46.012413 4717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "cfd1b9a59e8b42b8ac6673daff3b190d"
format_stamp: "Formatted at 2026-09-27 06:54:46 on dist-test-slave-f892"
server_key: "a63385c84414a31bc9d83b461789a7ef"
server_key_iv: "e7ae2d664928708d74b3be877bd138c3"
server_key_version: "encryptionkey@0"
I20260927 06:54:46.016996 4717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 06:54:46.020108 4731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:46.021265 4717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 06:54:46.021443 4717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "cfd1b9a59e8b42b8ac6673daff3b190d"
format_stamp: "Formatted at 2026-09-27 06:54:46 on dist-test-slave-f892"
server_key: "a63385c84414a31bc9d83b461789a7ef"
server_key_iv: "e7ae2d664928708d74b3be877bd138c3"
server_key_version: "encryptionkey@0"
I20260927 06:54:46.021643 4717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:46.044612 4717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:46.048576 4717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:46.048892 4717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:46.049860 4717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:46.051206 4717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:46.051334 4717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:46.051466 4717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:46.051558 4717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:46.083536 4717 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45919
I20260927 06:54:46.083597 4843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45919 every 8 connection(s)
I20260927 06:54:46.084487 4717 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:54:46.089547 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 4717
I20260927 06:54:46.089745 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:54:46.090241 388 external_mini_cluster.cc:1428] Setting key 8c19afe26e3e8931e3f2116c3da38dc5
Sep 27 06:54:46 dist-test-slave-f892 krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:46.106107 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:46.086894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42021 (local address 127.0.97.62:43517)
0927 06:54:46.087064 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:46.087068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:46.088114 (+ 1046us) server_negotiation.cc:407] Connection header received
0927 06:54:46.090062 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:46.090067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:46.090152 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:46.090264 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:46.091810 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.092613 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.095157 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.095551 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.095645 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:46.099524 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:46.099532 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:46.099540 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:46.099581 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:46.103105 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:46.103734 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:46.103742 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:46.103748 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:46.103910 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:46.104283 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:46.104291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:46.104297 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:46.104652 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:46.104803 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:46.105395 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 06:54:46.105640 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 06:54:46.106843 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:46.087309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43517 (local address 127.0.97.2:42021)
0927 06:54:46.087931 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:46.088004 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:46.089760 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:46.090475 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:46.090482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:46.091041 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:46.091615 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:46.091631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.092757 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.092765 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:46.095028 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:46.095045 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.095762 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.095772 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:46.096086 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:46.096795 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:46.096828 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:46.099316 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:46.103224 (+ 3908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:46.103230 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:46.103241 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:46.103618 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:46.104023 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:46.104026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:46.104029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:46.104182 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:46.104767 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:46.104773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:46.105048 (+ 275us) client_negotiation.cc:769] Sending connection context
0927 06:54:46.105259 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 06:54:46.105598 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":249,"threads_started":1}
I20260927 06:54:46.108258 4844 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43517
I20260927 06:54:46.108582 4844 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:46.109344 4844 heartbeater.cc:511] Master 127.0.97.62:43517 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 06:54:46.110914 4523 ts_manager.cc:194] Registered new tserver with Master: cfd1b9a59e8b42b8ac6673daff3b190d (127.0.97.2:45919)
I20260927 06:54:46.111706 4523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 06:54:46.134550 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43517
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:46.444229 4849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:46.444545 4849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:46.444717 4849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:46.451812 4849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:46.451951 4849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:46.452134 4849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:54:46.461540 4849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:46.462849 4849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:46.464136 4849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:46.476414 4855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:46.476478 4854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:46.476508 4857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:46.477303 4849 server_base.cc:1034] running on GCE node
I20260927 06:54:46.478092 4849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:46.478950 4849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:46.480175 4849 hybrid_clock.cc:648] HybridClock initialized: now 1790492086480131 us; error 52 us; skew 500 ppm
I20260927 06:54:46.480469 4849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:46 dist-test-slave-f892 krb5kdc[4479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:46.483422 4849 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:54:46.484499 4849 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:46.484620 4849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:46.484952 4849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:46.487720 4849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "5be6846e67ed4fb89bb4982b19bfacad"
format_stamp: "Formatted at 2026-09-27 06:54:46 on dist-test-slave-f892"
server_key: "682f0c3059a49e90d64096ec87b073e6"
server_key_iv: "899523eb6a257424a5f11ed3f2de3605"
server_key_version: "encryptionkey@0"
I20260927 06:54:46.488494 4849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5be6846e67ed4fb89bb4982b19bfacad"
format_stamp: "Formatted at 2026-09-27 06:54:46 on dist-test-slave-f892"
server_key: "682f0c3059a49e90d64096ec87b073e6"
server_key_iv: "899523eb6a257424a5f11ed3f2de3605"
server_key_version: "encryptionkey@0"
I20260927 06:54:46.493314 4849 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 06:54:46.496503 4863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:46.497638 4849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 06:54:46.497818 4849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "5be6846e67ed4fb89bb4982b19bfacad"
format_stamp: "Formatted at 2026-09-27 06:54:46 on dist-test-slave-f892"
server_key: "682f0c3059a49e90d64096ec87b073e6"
server_key_iv: "899523eb6a257424a5f11ed3f2de3605"
server_key_version: "encryptionkey@0"
I20260927 06:54:46.498050 4849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:46.519201 4849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:46.522969 4849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:46.523237 4849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:46.524132 4849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:46.525507 4849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:46.525624 4849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:46.525748 4849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:46.525872 4849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:46.557972 4849 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36275
I20260927 06:54:46.558079 4975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36275 every 8 connection(s)
I20260927 06:54:46.558910 4849 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:54:46.560838 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 4849
I20260927 06:54:46.561033 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:54:46.561565 388 external_mini_cluster.cc:1428] Setting key 4205261a738eb4bafc6abcc6ad9a59cc
Sep 27 06:54:46 dist-test-slave-f892 krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:46.577979 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:46.561417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38965 (local address 127.0.97.62:43517)
0927 06:54:46.561618 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:46.561624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:46.563221 (+ 1597us) server_negotiation.cc:407] Connection header received
0927 06:54:46.565342 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:46.565345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:46.565405 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:46.565490 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:46.566911 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.567611 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.569040 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.569410 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.569499 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:46.572571 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:46.572575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:46.572579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:46.572607 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:46.575277 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:46.575975 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:46.575981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:46.575984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:46.576101 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:46.576562 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:46.576567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:46.576570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:46.576870 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:46.576981 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:46.577552 (+ 571us) server_negotiation.cc:299] Negotiation successful
0927 06:54:46.577701 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 06:54:46.578835 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:46.561922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43517 (local address 127.0.97.3:38965)
0927 06:54:46.563048 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:46.563098 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:46.565115 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:46.565677 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:46.565684 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:46.566248 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:46.566778 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:46.566794 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.567729 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.567732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:46.568927 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:46.568938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.569591 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.569596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:46.569766 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:46.570357 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:46.570377 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:46.572394 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:46.575411 (+ 3017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:46.575419 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:46.575434 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:46.575853 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:46.576232 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:46.576235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:46.576238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:46.576427 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:46.576982 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:46.576988 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:46.577252 (+ 264us) client_negotiation.cc:769] Sending connection context
0927 06:54:46.577454 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 06:54:46.577792 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"thread_start_us":639,"threads_started":1}
I20260927 06:54:46.580149 4976 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43517
I20260927 06:54:46.580384 4976 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:46.580901 4976 heartbeater.cc:511] Master 127.0.97.62:43517 requested a full tablet report, sending...
I20260927 06:54:46.582185 4523 ts_manager.cc:194] Registered new tserver with Master: 5be6846e67ed4fb89bb4982b19bfacad (127.0.97.3:36275)
I20260927 06:54:46.582865 4523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38965
I20260927 06:54:46.588404 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:54:46.604909 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:46.592806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.0.97.62:43517)
0927 06:54:46.592985 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:46.592993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:46.593465 (+ 472us) server_negotiation.cc:407] Connection header received
0927 06:54:46.593672 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:46.593677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:46.593744 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:46.593843 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:46.595282 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.595960 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.597153 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.597590 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.597721 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:46.599861 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:46.599867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:46.599872 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:46.599909 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:46.602448 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:46.603056 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:46.603063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:46.603067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:46.603178 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:46.603536 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:46.603541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:46.603544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:46.603836 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:46.604088 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:46.604418 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 06:54:46.604622 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 06:54:46.608541 4523 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:54:46.615043 4523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:54:46.642593 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:46.626111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36275 (local address 127.0.0.1:50366)
0927 06:54:46.627187 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:46.627240 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:46.627431 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:46.628839 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:46.628849 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:46.628891 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:46.632029 (+ 3138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:46.632051 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.635373 (+ 3322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.635383 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:46.638450 (+ 3067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:46.638463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.641262 (+ 2799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.641275 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:46.641575 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:46.641650 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 06:54:46.641802 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 06:54:46.642072 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":6439,"threads_started":1}
I20260927 06:54:46.644671 4711 heartbeater.cc:503] Master 127.0.97.62:43517 was elected leader, sending a full tablet report...
I20260927 06:54:46.645656 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:46.626861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50366 (local address 127.0.97.3:36275)
0927 06:54:46.628102 (+ 1241us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:46.628114 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:46.628154 (+ 40us) server_negotiation.cc:407] Connection header received
0927 06:54:46.628262 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:46.628271 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:46.628572 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:46.632966 (+ 4394us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:46.633348 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.635183 (+ 1835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.638658 (+ 3475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.640615 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.640785 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:46.641100 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:46.644301 (+ 3201us) server_negotiation.cc:299] Negotiation successful
0927 06:54:46.645233 (+ 932us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1037,"thread_start_us":1530,"threads_started":1}
I20260927 06:54:46.649781 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:46.626358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35595 (local address 127.0.0.1:53666)
0927 06:54:46.632883 (+ 6525us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:46.632907 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:46.633052 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:46.640787 (+ 7735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:46.640795 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:46.640832 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:46.641407 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:46.641422 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.643834 (+ 2412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.643840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:46.646023 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:46.646037 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.649029 (+ 2992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.649036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:46.649225 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:46.649256 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 06:54:46.649318 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 06:54:46.649438 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6352,"thread_start_us":674,"threads_started":1}
I20260927 06:54:46.651575 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:46.626668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53666 (local address 127.0.97.1:35595)
0927 06:54:46.639713 (+ 13045us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:46.639726 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:46.639764 (+ 38us) server_negotiation.cc:407] Connection header received
0927 06:54:46.639895 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:46.639907 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:46.640267 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:46.641667 (+ 1400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:46.641929 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.643678 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.647419 (+ 3741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.648797 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.650867 (+ 2070us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:46.651064 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:46.651128 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 06:54:46.651273 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12810,"thread_start_us":13785,"threads_started":1}
I20260927 06:54:46.650866 4911 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf928ad9cf84fb083f12edc0819c4d6 (DEFAULT_TABLE table=test-table [id=32dfc6f7ef534adb85910f5bb202f7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:46.651763 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:46.629602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45919 (local address 127.0.0.1:39694)
0927 06:54:46.643911 (+ 14309us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:46.643935 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:46.644086 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:46.644875 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:46.644880 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:46.644904 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:46.645063 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:46.645072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.646890 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.646894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:46.649043 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:46.649072 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.651049 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.651054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:46.651222 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:46.651269 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 06:54:46.651354 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 06:54:46.651492 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14179,"threads_started":1}
I20260927 06:54:46.652362 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:46.629689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39694 (local address 127.0.97.2:45919)
0927 06:54:46.643324 (+ 13635us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:46.643331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:46.644044 (+ 713us) server_negotiation.cc:407] Connection header received
0927 06:54:46.644260 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:46.644268 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:46.644486 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:46.644664 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:46.645215 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.646714 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.649222 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.650646 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.650766 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:46.650955 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:46.651925 (+ 970us) server_negotiation.cc:299] Negotiation successful
0927 06:54:46.652072 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13479,"thread_start_us":13382,"threads_started":1}
I20260927 06:54:46.654487 4911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf928ad9cf84fb083f12edc0819c4d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:46.655113 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf928ad9cf84fb083f12edc0819c4d6 (DEFAULT_TABLE table=test-table [id=32dfc6f7ef534adb85910f5bb202f7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:46.656366 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf928ad9cf84fb083f12edc0819c4d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:46.655926 4779 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf928ad9cf84fb083f12edc0819c4d6 (DEFAULT_TABLE table=test-table [id=32dfc6f7ef534adb85910f5bb202f7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:46.657111 4779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf928ad9cf84fb083f12edc0819c4d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:46.670486 4995 tablet_bootstrap.cc:492] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c: Bootstrap starting.
I20260927 06:54:46.674098 4994 tablet_bootstrap.cc:492] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad: Bootstrap starting.
I20260927 06:54:46.674826 4995 tablet_bootstrap.cc:654] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:46.675868 4996 tablet_bootstrap.cc:492] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d: Bootstrap starting.
I20260927 06:54:46.676759 4995 log.cc:824] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:46.678645 4994 tablet_bootstrap.cc:654] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:46.679219 4996 tablet_bootstrap.cc:654] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:46.679702 4994 log.cc:824] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:46.680198 4995 tablet_bootstrap.cc:492] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c: No bootstrap required, opened a new log
I20260927 06:54:46.680377 4996 log.cc:824] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:46.680534 4995 ts_tablet_manager.cc:1397] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260927 06:54:46.682415 4994 tablet_bootstrap.cc:492] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad: No bootstrap required, opened a new log
I20260927 06:54:46.682677 4994 ts_tablet_manager.cc:1397] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260927 06:54:46.684504 4996 tablet_bootstrap.cc:492] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d: No bootstrap required, opened a new log
I20260927 06:54:46.684783 4996 ts_tablet_manager.cc:1397] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 06:54:46.690013 4994 raft_consensus.cc:348] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be6846e67ed4fb89bb4982b19bfacad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36275 } } peers { permanent_uuid: "7e24ec310e404d0a9f5dbde7b5f3f10c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35595 } } peers { permanent_uuid: "cfd1b9a59e8b42b8ac6673daff3b190d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } }
I20260927 06:54:46.690575 4994 raft_consensus.cc:374] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:46.690729 4994 raft_consensus.cc:729] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5be6846e67ed4fb89bb4982b19bfacad, State: Initialized, Role: FOLLOWER
I20260927 06:54:46.691154 4995 raft_consensus.cc:348] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be6846e67ed4fb89bb4982b19bfacad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36275 } } peers { permanent_uuid: "7e24ec310e404d0a9f5dbde7b5f3f10c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35595 } } peers { permanent_uuid: "cfd1b9a59e8b42b8ac6673daff3b190d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } }
I20260927 06:54:46.691557 4995 raft_consensus.cc:374] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:46.691433 4994 consensus_queue.cc:260] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be6846e67ed4fb89bb4982b19bfacad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36275 } } peers { permanent_uuid: "7e24ec310e404d0a9f5dbde7b5f3f10c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35595 } } peers { permanent_uuid: "cfd1b9a59e8b42b8ac6673daff3b190d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } }
I20260927 06:54:46.691648 4995 raft_consensus.cc:729] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e24ec310e404d0a9f5dbde7b5f3f10c, State: Initialized, Role: FOLLOWER
I20260927 06:54:46.692180 4995 consensus_queue.cc:260] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be6846e67ed4fb89bb4982b19bfacad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36275 } } peers { permanent_uuid: "7e24ec310e404d0a9f5dbde7b5f3f10c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35595 } } peers { permanent_uuid: "cfd1b9a59e8b42b8ac6673daff3b190d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } }
I20260927 06:54:46.693929 4995 ts_tablet_manager.cc:1428] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260927 06:54:46.695073 4976 heartbeater.cc:503] Master 127.0.97.62:43517 was elected leader, sending a full tablet report...
I20260927 06:54:46.695721 4996 raft_consensus.cc:348] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be6846e67ed4fb89bb4982b19bfacad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36275 } } peers { permanent_uuid: "7e24ec310e404d0a9f5dbde7b5f3f10c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35595 } } peers { permanent_uuid: "cfd1b9a59e8b42b8ac6673daff3b190d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } }
I20260927 06:54:46.696069 4996 raft_consensus.cc:374] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:46.696157 4996 raft_consensus.cc:729] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd1b9a59e8b42b8ac6673daff3b190d, State: Initialized, Role: FOLLOWER
I20260927 06:54:46.697350 4994 ts_tablet_manager.cc:1428] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s
I20260927 06:54:46.698112 4996 consensus_queue.cc:260] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be6846e67ed4fb89bb4982b19bfacad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36275 } } peers { permanent_uuid: "7e24ec310e404d0a9f5dbde7b5f3f10c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35595 } } peers { permanent_uuid: "cfd1b9a59e8b42b8ac6673daff3b190d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } }
I20260927 06:54:46.699218 4844 heartbeater.cc:503] Master 127.0.97.62:43517 was elected leader, sending a full tablet report...
I20260927 06:54:46.699718 4996 ts_tablet_manager.cc:1428] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260927 06:54:46.705426 5001 raft_consensus.cc:482] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:54:46.705701 5001 raft_consensus.cc:504] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be6846e67ed4fb89bb4982b19bfacad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36275 } } peers { permanent_uuid: "7e24ec310e404d0a9f5dbde7b5f3f10c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35595 } } peers { permanent_uuid: "cfd1b9a59e8b42b8ac6673daff3b190d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } }
I20260927 06:54:46.707424 5001 leader_election.cc:288] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5be6846e67ed4fb89bb4982b19bfacad (127.0.97.3:36275), cfd1b9a59e8b42b8ac6673daff3b190d (127.0.97.2:45919)
I20260927 06:54:46.715688 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:46.707685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45919 (local address 127.0.97.1:56175)
0927 06:54:46.708756 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:46.708784 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:46.708941 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:46.709292 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:46.709295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:46.709311 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:46.709850 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:46.709868 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.711273 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.711278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:46.713971 (+ 2693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:46.713980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.715083 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.715088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:46.715202 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:46.715226 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 06:54:46.715307 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 06:54:46.715395 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"mutex_wait_us":198,"spinlock_wait_cycles":33536,"thread_start_us":1073,"threads_started":1}
I20260927 06:54:46.715749 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:46.707944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56175 (local address 127.0.97.2:45919)
0927 06:54:46.708197 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:46.708205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:46.708889 (+ 684us) server_negotiation.cc:407] Connection header received
0927 06:54:46.709075 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:46.709079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:46.709160 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:46.709258 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:46.710056 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.711130 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.714125 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.714898 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.715008 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:46.715081 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:46.715360 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 06:54:46.715499 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 06:54:46.716771 4799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf928ad9cf84fb083f12edc0819c4d6" candidate_uuid: "7e24ec310e404d0a9f5dbde7b5f3f10c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfd1b9a59e8b42b8ac6673daff3b190d" is_pre_election: true
I20260927 06:54:46.717015 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:46.707786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56995 (local address 127.0.97.3:36275)
0927 06:54:46.708013 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:46.708022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:46.709154 (+ 1132us) server_negotiation.cc:407] Connection header received
0927 06:54:46.709251 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:46.709256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:46.709321 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:46.709413 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:54:46.710069 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.711143 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.715147 (+ 4004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.716250 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.716344 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:46.716412 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:46.716764 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 06:54:46.716853 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 06:54:46.717234 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:46.707713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36275 (local address 127.0.97.1:56995)
0927 06:54:46.709054 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:46.709071 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:46.709163 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:46.709435 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:46.709444 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:46.709481 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:54:46.709899 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:46.709909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.711274 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.711278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:46.713965 (+ 2687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:46.713980 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.716426 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.716432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:46.716607 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:46.716644 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 06:54:46.716740 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 06:54:46.716871 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"mutex_wait_us":138,"thread_start_us":689,"threads_started":1}
I20260927 06:54:46.717332 4799 raft_consensus.cc:2454] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e24ec310e404d0a9f5dbde7b5f3f10c in term 0.
I20260927 06:54:46.717985 4599 leader_election.cc:302] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e24ec310e404d0a9f5dbde7b5f3f10c, cfd1b9a59e8b42b8ac6673daff3b190d; no voters:
I20260927 06:54:46.718144 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf928ad9cf84fb083f12edc0819c4d6" candidate_uuid: "7e24ec310e404d0a9f5dbde7b5f3f10c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5be6846e67ed4fb89bb4982b19bfacad" is_pre_election: true
I20260927 06:54:46.718430 5001 raft_consensus.cc:2781] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:54:46.718588 5001 raft_consensus.cc:482] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:54:46.718680 5001 raft_consensus.cc:3037] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:46.718680 4931 raft_consensus.cc:2454] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e24ec310e404d0a9f5dbde7b5f3f10c in term 0.
I20260927 06:54:46.721048 5001 raft_consensus.cc:504] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be6846e67ed4fb89bb4982b19bfacad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36275 } } peers { permanent_uuid: "7e24ec310e404d0a9f5dbde7b5f3f10c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35595 } } peers { permanent_uuid: "cfd1b9a59e8b42b8ac6673daff3b190d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } }
I20260927 06:54:46.721691 5001 leader_election.cc:288] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [CANDIDATE]: Term 1 election: Requested vote from peers 5be6846e67ed4fb89bb4982b19bfacad (127.0.97.3:36275), cfd1b9a59e8b42b8ac6673daff3b190d (127.0.97.2:45919)
I20260927 06:54:46.722054 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf928ad9cf84fb083f12edc0819c4d6" candidate_uuid: "7e24ec310e404d0a9f5dbde7b5f3f10c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5be6846e67ed4fb89bb4982b19bfacad"
I20260927 06:54:46.722254 4931 raft_consensus.cc:3037] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:46.722244 4799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf928ad9cf84fb083f12edc0819c4d6" candidate_uuid: "7e24ec310e404d0a9f5dbde7b5f3f10c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfd1b9a59e8b42b8ac6673daff3b190d"
I20260927 06:54:46.722432 4799 raft_consensus.cc:3037] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:46.725411 4799 raft_consensus.cc:2454] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e24ec310e404d0a9f5dbde7b5f3f10c in term 1.
I20260927 06:54:46.725409 4931 raft_consensus.cc:2454] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e24ec310e404d0a9f5dbde7b5f3f10c in term 1.
I20260927 06:54:46.725746 4599 leader_election.cc:302] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e24ec310e404d0a9f5dbde7b5f3f10c, cfd1b9a59e8b42b8ac6673daff3b190d; no voters:
I20260927 06:54:46.726052 5001 raft_consensus.cc:2781] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:46.726471 5001 raft_consensus.cc:686] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [term 1 LEADER]: Becoming Leader. State: Replica: 7e24ec310e404d0a9f5dbde7b5f3f10c, State: Running, Role: LEADER
I20260927 06:54:46.726936 5001 consensus_queue.cc:237] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be6846e67ed4fb89bb4982b19bfacad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36275 } } peers { permanent_uuid: "7e24ec310e404d0a9f5dbde7b5f3f10c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35595 } } peers { permanent_uuid: "cfd1b9a59e8b42b8ac6673daff3b190d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } }
I20260927 06:54:46.731148 4523 catalog_manager.cc:5495] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e24ec310e404d0a9f5dbde7b5f3f10c (127.0.97.1). New cstate: current_term: 1 leader_uuid: "7e24ec310e404d0a9f5dbde7b5f3f10c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be6846e67ed4fb89bb4982b19bfacad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e24ec310e404d0a9f5dbde7b5f3f10c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfd1b9a59e8b42b8ac6673daff3b190d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } health_report { overall_health: UNKNOWN } } }
I20260927 06:54:46.761016 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:46.755762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.0.97.1:35595)
0927 06:54:46.755976 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:46.755985 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:46.756046 (+ 61us) server_negotiation.cc:407] Connection header received
0927 06:54:46.756287 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:46.756291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:46.756371 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:46.756524 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:54:46.756879 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.757595 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.758991 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.759381 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.759514 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:46.759979 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:54:46.760461 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:54:46.760617 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:46.760671 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 06:54:46.760746 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 06:54:46.768343 4799 raft_consensus.cc:1261] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d [term 1 FOLLOWER]: Refusing update from remote peer 7e24ec310e404d0a9f5dbde7b5f3f10c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:54:46.768383 4931 raft_consensus.cc:1261] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad [term 1 FOLLOWER]: Refusing update from remote peer 7e24ec310e404d0a9f5dbde7b5f3f10c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:54:46.769244 5005 consensus_queue.cc:1035] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5be6846e67ed4fb89bb4982b19bfacad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:46.769573 5001 consensus_queue.cc:1035] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfd1b9a59e8b42b8ac6673daff3b190d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:46.782788 5009 mvcc.cc:205] Tried to move back new op lower bound from 7333855587394744320 to 7333855587236794368. Current Snapshot: MvccSnapshot[applied={T|T < 7333855587394744320}]
I20260927 06:54:46.794274 4523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:54:46.794564 4523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:54:46.798108 4523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a52da375a2f44d64a4d3295285aa6625: Sending DeleteTablet for 3 replicas of tablet 8bf928ad9cf84fb083f12edc0819c4d6
I20260927 06:54:46.799191 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf928ad9cf84fb083f12edc0819c4d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53666
I20260927 06:54:46.799185 4911 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf928ad9cf84fb083f12edc0819c4d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50366
I20260927 06:54:46.799328 4779 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf928ad9cf84fb083f12edc0819c4d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39694
I20260927 06:54:46.807934 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 4584
I20260927 06:54:46.811501 5019 tablet_replica.cc:330] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d: stopping tablet replica
I20260927 06:54:46.812070 5019 raft_consensus.cc:2229] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:54:46.812662 5019 raft_consensus.cc:2258] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:46.816715 5019 ts_tablet_manager.cc:1905] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:46.824664 5019 ts_tablet_manager.cc:1918] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:54:46.824891 5019 log.cc:1196] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/8bf928ad9cf84fb083f12edc0819c4d6
I20260927 06:54:46.825539 5019 ts_tablet_manager.cc:1939] T 8bf928ad9cf84fb083f12edc0819c4d6 P cfd1b9a59e8b42b8ac6673daff3b190d: Deleting consensus metadata
I20260927 06:54:46.827387 4507 catalog_manager.cc:4841] TS cfd1b9a59e8b42b8ac6673daff3b190d (127.0.97.2:45919): tablet 8bf928ad9cf84fb083f12edc0819c4d6 (table test-table [id=32dfc6f7ef534adb85910f5bb202f7ff]) successfully deleted
I20260927 06:54:46.827647 5018 tablet_replica.cc:330] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c: stopping tablet replica
I20260927 06:54:46.828055 5018 raft_consensus.cc:2229] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:54:46.828743 5018 raft_consensus.cc:2258] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:46.829558 5017 tablet_replica.cc:330] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad: stopping tablet replica
I20260927 06:54:46.829916 5017 raft_consensus.cc:2229] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:54:46.830888 5017 raft_consensus.cc:2258] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:46.831478 5018 ts_tablet_manager.cc:1905] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:46.832895 5017 ts_tablet_manager.cc:1905] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 06:54:46 dist-test-slave-f892 krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:54:46.839063 5018 ts_tablet_manager.cc:1918] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:54:46.839334 5018 log.cc:1196] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/8bf928ad9cf84fb083f12edc0819c4d6
I20260927 06:54:46.839901 5018 ts_tablet_manager.cc:1939] T 8bf928ad9cf84fb083f12edc0819c4d6 P 7e24ec310e404d0a9f5dbde7b5f3f10c: Deleting consensus metadata
I20260927 06:54:46.840301 5017 ts_tablet_manager.cc:1918] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:54:46.840493 5017 log.cc:1196] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/8bf928ad9cf84fb083f12edc0819c4d6
I20260927 06:54:46.841063 5017 ts_tablet_manager.cc:1939] T 8bf928ad9cf84fb083f12edc0819c4d6 P 5be6846e67ed4fb89bb4982b19bfacad: Deleting consensus metadata
I20260927 06:54:46.841764 4510 catalog_manager.cc:4841] TS 7e24ec310e404d0a9f5dbde7b5f3f10c (127.0.97.1:35595): tablet 8bf928ad9cf84fb083f12edc0819c4d6 (table test-table [id=32dfc6f7ef534adb85910f5bb202f7ff]) successfully deleted
I20260927 06:54:46.842772 4509 catalog_manager.cc:4841] TS 5be6846e67ed4fb89bb4982b19bfacad (127.0.97.3:36275): tablet 8bf928ad9cf84fb083f12edc0819c4d6 (table test-table [id=32dfc6f7ef534adb85910f5bb202f7ff]) successfully deleted
I20260927 06:54:46.843147 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:46.809541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.0.97.1:35595)
0927 06:54:46.809756 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:46.809765 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:46.822914 (+ 13149us) server_negotiation.cc:407] Connection header received
0927 06:54:46.823354 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:46.823360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:46.823438 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:46.823546 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:46.825432 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.826700 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.828548 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.829282 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.829382 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:46.835317 (+ 5935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:46.835332 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:46.835349 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:46.835400 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:46.839507 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:46.840338 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:46.840352 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:46.840356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:46.840548 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:46.841010 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:46.841015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:46.841020 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:46.841476 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:46.841759 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:46.842324 (+ 565us) server_negotiation.cc:299] Negotiation successful
0927 06:54:46.842558 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 06:54:46.843786 4706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:46.941890 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 4584
I20260927 06:54:46.959254 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 4717
Sep 27 06:54:46 dist-test-slave-f892 krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:54:46.972536 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:46.960063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.0.97.2:45919)
0927 06:54:46.960222 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:46.960226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:46.960244 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:54:46.960352 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:46.960356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:46.960444 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:46.960530 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:46.961606 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.962382 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.963484 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:46.963850 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:46.963939 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:46.967677 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:46.967688 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:46.967702 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:46.967733 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:46.970391 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:46.970909 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:46.970914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:46.970917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:46.971028 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:46.971305 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:46.971308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:46.971310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:46.971640 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:46.971837 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:46.972020 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 06:54:46.972158 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 06:54:46.973188 4839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:47.059581 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 4717
I20260927 06:54:47.075738 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 4849
Sep 27 06:54:47 dist-test-slave-f892 krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 06:54:47.090157 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:47.076473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.0.97.3:36275)
0927 06:54:47.076647 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:47.076653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:47.076715 (+ 62us) server_negotiation.cc:407] Connection header received
0927 06:54:47.076906 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:47.076912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:47.076994 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:47.077099 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:47.078387 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:47.079181 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:47.080419 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:47.080800 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:47.080929 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:47.085090 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:47.085103 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:47.085118 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:47.085153 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:47.087658 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:47.088323 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:47.088330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:47.088332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:47.088443 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:47.088744 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:47.088746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:47.088748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:47.089141 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:47.089329 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:47.089585 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 06:54:47.089739 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 06:54:47.090780 4971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:47.176565 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 4849
I20260927 06:54:47.192970 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 4492
I20260927 06:54:47.194146 4553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:47.299675 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 4492
2026-09-27T06:54:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3948 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T06:54:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:54:47Z Disabled control of system clock
I20260927 06:54:47.364347 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33573
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 06:54:47.671263 5028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:47.671552 5028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:47.671635 5028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:47.679453 5028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:54:47.679551 5028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:47.679587 5028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:47.679641 5028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:54:47.679682 5028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:54:47.688611 5028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33573
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:47.689795 5028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:47.691447 5028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:47.702564 5033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:47.702533 5036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:47.702595 5034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:47.703330 5028 server_base.cc:1034] running on GCE node
I20260927 06:54:47.704097 5028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:47.705379 5028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:47.706624 5028 hybrid_clock.cc:648] HybridClock initialized: now 1790492087706601 us; error 63 us; skew 500 ppm
I20260927 06:54:47.706954 5028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:54:47.709026 5028 webserver.cc:460] Webserver started at http://127.0.97.62:44319/ using document root <none> and password file <none>
I20260927 06:54:47.709667 5028 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:47.709790 5028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:47.710183 5028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:47.712908 5028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "7b18c6a829e140a381d8c37e256e82cb"
format_stamp: "Formatted at 2026-09-27 06:54:47 on dist-test-slave-f892"
server_key: "fd8fd67251bbc2a8fbdd080be2bb6317"
server_key_iv: "79b4cb40eeac74f895b4a3a61057d952"
server_key_version: "encryptionkey@0"
I20260927 06:54:47.713661 5028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "7b18c6a829e140a381d8c37e256e82cb"
format_stamp: "Formatted at 2026-09-27 06:54:47 on dist-test-slave-f892"
server_key: "fd8fd67251bbc2a8fbdd080be2bb6317"
server_key_iv: "79b4cb40eeac74f895b4a3a61057d952"
server_key_version: "encryptionkey@0"
I20260927 06:54:47.718389 5028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260927 06:54:47.721452 5042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:47.722671 5028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 06:54:47.722836 5028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "7b18c6a829e140a381d8c37e256e82cb"
format_stamp: "Formatted at 2026-09-27 06:54:47 on dist-test-slave-f892"
server_key: "fd8fd67251bbc2a8fbdd080be2bb6317"
server_key_iv: "79b4cb40eeac74f895b4a3a61057d952"
server_key_version: "encryptionkey@0"
I20260927 06:54:47.723057 5028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:47.728293 5028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:47.728971 5028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:47.729238 5028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:47.746937 5028 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33573
I20260927 06:54:47.746977 5093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33573 every 8 connection(s)
I20260927 06:54:47.748171 5028 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:54:47.749945 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 5028
I20260927 06:54:47.750164 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:54:47.750798 388 external_mini_cluster.cc:1428] Setting key d7a5fc587b91e882d1f72221c891493d
I20260927 06:54:47.752177 5094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:47.759352 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:47.752817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.0.97.62:33573)
0927 06:54:47.753855 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:47.753868 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:47.753916 (+ 48us) server_negotiation.cc:407] Connection header received
0927 06:54:47.755581 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:47.755604 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:47.756100 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:47.756491 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:47.757472 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:47.757485 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:47.757502 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:47.757518 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:47.757598 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:47.757622 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:47.757827 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:47.757942 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 06:54:47.758237 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":347,"threads_started":1}
I20260927 06:54:47.764752 5094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb: Bootstrap starting.
I20260927 06:54:47.767438 5094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:47.768313 5094 log.cc:824] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:47.770646 5094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb: No bootstrap required, opened a new log
I20260927 06:54:47.777074 5094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b18c6a829e140a381d8c37e256e82cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33573 } }
I20260927 06:54:47.777344 5094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:47.777477 5094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b18c6a829e140a381d8c37e256e82cb, State: Initialized, Role: FOLLOWER
I20260927 06:54:47.777993 5094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b18c6a829e140a381d8c37e256e82cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33573 } }
I20260927 06:54:47.778221 5094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:54:47.778313 5094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:54:47.778427 5094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:47.780315 5094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b18c6a829e140a381d8c37e256e82cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33573 } }
I20260927 06:54:47.780712 5094 leader_election.cc:302] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b18c6a829e140a381d8c37e256e82cb; no voters:
I20260927 06:54:47.781157 5094 leader_election.cc:288] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:54:47.781292 5099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:47.781970 5099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb [term 1 LEADER]: Becoming Leader. State: Replica: 7b18c6a829e140a381d8c37e256e82cb, State: Running, Role: LEADER
I20260927 06:54:47.782502 5099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b18c6a829e140a381d8c37e256e82cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33573 } }
I20260927 06:54:47.782967 5094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:54:47.784863 5100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b18c6a829e140a381d8c37e256e82cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b18c6a829e140a381d8c37e256e82cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33573 } } }
I20260927 06:54:47.785230 5100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:47.785157 5101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b18c6a829e140a381d8c37e256e82cb. Latest consensus state: current_term: 1 leader_uuid: "7b18c6a829e140a381d8c37e256e82cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b18c6a829e140a381d8c37e256e82cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33573 } } }
I20260927 06:54:47.785456 5101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:47.788223 5104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:54:47.792538 5104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:54:47.800442 5104 catalog_manager.cc:1347] Generated new cluster ID: 15ea156024104f9394ebea8e45074f51
I20260927 06:54:47.800556 5104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:54:47.809966 5104 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:54:47.810379 5104 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:54:47.818663 5104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb: Generated new TSK 0
I20260927 06:54:47.819495 5104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:54:47.829612 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33573
--builtin_ntp_servers=127.0.97.20:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 06:54:48.146201 5118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:48.146493 5118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:48.146656 5118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:48.153647 5118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:48.153782 5118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:48.153983 5118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:54:48.162946 5118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:48.164222 5118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:48.165999 5118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:48.178237 5123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:48.178208 5126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:48.178385 5124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:48.179469 5118 server_base.cc:1034] running on GCE node
I20260927 06:54:48.180073 5118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:48.181412 5118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:48.182629 5118 hybrid_clock.cc:648] HybridClock initialized: now 1790492088182594 us; error 46 us; skew 500 ppm
I20260927 06:54:48.182934 5118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:54:48.188315 5118 webserver.cc:460] Webserver started at http://127.0.97.1:35703/ using document root <none> and password file <none>
I20260927 06:54:48.188948 5118 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:48.189077 5118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:48.189380 5118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:48.192152 5118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "ee037ebd5bbf44bfa4c0191ddbec7bf3"
format_stamp: "Formatted at 2026-09-27 06:54:48 on dist-test-slave-f892"
server_key: "32cada5b5d09d7f64aa88a49f69e038d"
server_key_iv: "be8eed16229abaa271ab2b7a541e1d71"
server_key_version: "encryptionkey@0"
I20260927 06:54:48.192939 5118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ee037ebd5bbf44bfa4c0191ddbec7bf3"
format_stamp: "Formatted at 2026-09-27 06:54:48 on dist-test-slave-f892"
server_key: "32cada5b5d09d7f64aa88a49f69e038d"
server_key_iv: "be8eed16229abaa271ab2b7a541e1d71"
server_key_version: "encryptionkey@0"
I20260927 06:54:48.197708 5118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 06:54:48.200886 5132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:48.202185 5118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 06:54:48.202345 5118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "ee037ebd5bbf44bfa4c0191ddbec7bf3"
format_stamp: "Formatted at 2026-09-27 06:54:48 on dist-test-slave-f892"
server_key: "32cada5b5d09d7f64aa88a49f69e038d"
server_key_iv: "be8eed16229abaa271ab2b7a541e1d71"
server_key_version: "encryptionkey@0"
I20260927 06:54:48.202579 5118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:48.207818 5118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:48.208564 5118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:48.208820 5118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:48.209647 5118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:48.211004 5118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:48.211120 5118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:48.211247 5118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:48.211331 5118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:48.243294 5244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46163 every 8 connection(s)
I20260927 06:54:48.243302 5118 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46163
I20260927 06:54:48.244738 5118 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:54:48.246109 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 5118
I20260927 06:54:48.246304 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:54:48.247064 388 external_mini_cluster.cc:1428] Setting key 18e0f0717723fddc6082a063dcb429a7
I20260927 06:54:48.250079 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33573
--builtin_ntp_servers=127.0.97.20:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 06:54:48.253775 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:48.247365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58021 (local address 127.0.97.62:33573)
0927 06:54:48.247544 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:48.247549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:48.249094 (+ 1545us) server_negotiation.cc:407] Connection header received
0927 06:54:48.251690 (+ 2596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:48.251695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:48.251740 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:48.251827 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:48.253173 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:48.253177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:48.253180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:48.253195 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:48.253224 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:48.253235 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:48.253326 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:48.253552 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 06:54:48.253595 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 06:54:48.254962 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:48.248003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33573 (local address 127.0.97.1:58021)
0927 06:54:48.248952 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:48.248991 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:48.251415 (+ 2424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:48.252064 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:48.252077 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:48.252184 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:48.252859 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:54:48.252866 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:48.252914 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:54:48.253017 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:48.253329 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:48.253342 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:48.253353 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 06:54:48.253623 (+ 270us) client_negotiation.cc:240] Negotiation successful
0927 06:54:48.253838 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":286,"threads_started":1}
I20260927 06:54:48.256196 5245 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33573
I20260927 06:54:48.256531 5245 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:48.257243 5245 heartbeater.cc:511] Master 127.0.97.62:33573 requested a full tablet report, sending...
I20260927 06:54:48.259449 5059 ts_manager.cc:194] Registered new tserver with Master: ee037ebd5bbf44bfa4c0191ddbec7bf3 (127.0.97.1:46163)
W20260927 06:54:48.565889 5248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:48.566160 5248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:48.566248 5248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:48.573211 5248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:48.573302 5248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:48.573413 5248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:54:48.582180 5248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:48.583356 5248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:48.585069 5248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:48.597713 5253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:48.597667 5254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:48.598291 5256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:48.598739 5248 server_base.cc:1034] running on GCE node
I20260927 06:54:48.599327 5248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:48.600221 5248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:48.601501 5248 hybrid_clock.cc:648] HybridClock initialized: now 1790492088601440 us; error 54 us; skew 500 ppm
I20260927 06:54:48.601817 5248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:54:48.604020 5248 webserver.cc:460] Webserver started at http://127.0.97.2:43675/ using document root <none> and password file <none>
I20260927 06:54:48.604688 5248 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:48.604823 5248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:48.605140 5248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:48.608072 5248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "65ed0a2b135846dc8b70691a4441c98b"
format_stamp: "Formatted at 2026-09-27 06:54:48 on dist-test-slave-f892"
server_key: "0b54681c6b5218aac352f1c3285322bd"
server_key_iv: "876170bf045934727ae4b386af58bc45"
server_key_version: "encryptionkey@0"
I20260927 06:54:48.608847 5248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "65ed0a2b135846dc8b70691a4441c98b"
format_stamp: "Formatted at 2026-09-27 06:54:48 on dist-test-slave-f892"
server_key: "0b54681c6b5218aac352f1c3285322bd"
server_key_iv: "876170bf045934727ae4b386af58bc45"
server_key_version: "encryptionkey@0"
I20260927 06:54:48.613719 5248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 06:54:48.616878 5262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:48.618312 5248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:54:48.618505 5248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "65ed0a2b135846dc8b70691a4441c98b"
format_stamp: "Formatted at 2026-09-27 06:54:48 on dist-test-slave-f892"
server_key: "0b54681c6b5218aac352f1c3285322bd"
server_key_iv: "876170bf045934727ae4b386af58bc45"
server_key_version: "encryptionkey@0"
I20260927 06:54:48.618712 5248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:48.623642 5248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:48.624325 5248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:48.624578 5248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:48.625363 5248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:48.626893 5248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:48.627015 5248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:48.627137 5248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:48.627233 5248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:48.659030 5248 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34599
I20260927 06:54:48.659082 5374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34599 every 8 connection(s)
I20260927 06:54:48.660284 5248 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:54:48.660670 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 5248
I20260927 06:54:48.660844 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:54:48.661448 388 external_mini_cluster.cc:1428] Setting key 217e423641783280e978dbe902790897
I20260927 06:54:48.664476 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33573
--builtin_ntp_servers=127.0.97.20:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 06:54:48.668661 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:48.662860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55425 (local address 127.0.97.62:33573)
0927 06:54:48.663035 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:48.663041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:48.664716 (+ 1675us) server_negotiation.cc:407] Connection header received
0927 06:54:48.666464 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:48.666479 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:48.666545 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:48.666654 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:48.667892 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:48.667901 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:48.667909 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:48.667931 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:48.667969 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:48.667986 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:48.668103 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:48.668260 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 06:54:48.668323 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 06:54:48.669413 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:48.663546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33573 (local address 127.0.97.2:55425)
0927 06:54:48.664535 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:48.664608 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:48.666191 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:48.666842 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:48.666849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:48.666930 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:48.667605 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:54:48.667608 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:48.667642 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:54:48.667721 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:48.668080 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:48.668087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:48.668092 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 06:54:48.668286 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 06:54:48.668480 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":327,"threads_started":1}
I20260927 06:54:48.670851 5375 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33573
I20260927 06:54:48.671178 5375 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:48.671759 5375 heartbeater.cc:511] Master 127.0.97.62:33573 requested a full tablet report, sending...
I20260927 06:54:48.673224 5059 ts_manager.cc:194] Registered new tserver with Master: 65ed0a2b135846dc8b70691a4441c98b (127.0.97.2:34599)
W20260927 06:54:48.978052 5378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:48.978345 5378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:48.978513 5378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:48.985806 5378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:48.985924 5378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:48.986119 5378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:54:48.994846 5378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:48.996084 5378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:48.997776 5378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:49.010416 5386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:49.010460 5384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:49.010648 5383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:49.011322 5378 server_base.cc:1034] running on GCE node
I20260927 06:54:49.011847 5378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:49.012732 5378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:49.013947 5378 hybrid_clock.cc:648] HybridClock initialized: now 1790492089013905 us; error 54 us; skew 500 ppm
I20260927 06:54:49.014256 5378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:54:49.016266 5378 webserver.cc:460] Webserver started at http://127.0.97.3:43443/ using document root <none> and password file <none>
I20260927 06:54:49.016899 5378 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:49.017030 5378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:49.017329 5378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:49.020020 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "e3ed8678c7cf4d14ae8cef29c8577cbd"
format_stamp: "Formatted at 2026-09-27 06:54:49 on dist-test-slave-f892"
server_key: "0ce992a0eb6e46cc85c426fd5511e97a"
server_key_iv: "b1fa66fbb2f110e8ec5407b07c322b68"
server_key_version: "encryptionkey@0"
I20260927 06:54:49.020754 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e3ed8678c7cf4d14ae8cef29c8577cbd"
format_stamp: "Formatted at 2026-09-27 06:54:49 on dist-test-slave-f892"
server_key: "0ce992a0eb6e46cc85c426fd5511e97a"
server_key_iv: "b1fa66fbb2f110e8ec5407b07c322b68"
server_key_version: "encryptionkey@0"
I20260927 06:54:49.025403 5378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260927 06:54:49.028637 5392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:49.029832 5378 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 06:54:49.030109 5378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "e3ed8678c7cf4d14ae8cef29c8577cbd"
format_stamp: "Formatted at 2026-09-27 06:54:49 on dist-test-slave-f892"
server_key: "0ce992a0eb6e46cc85c426fd5511e97a"
server_key_iv: "b1fa66fbb2f110e8ec5407b07c322b68"
server_key_version: "encryptionkey@0"
I20260927 06:54:49.030401 5378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:49.035983 5378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:49.036787 5378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:49.037056 5378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:49.037933 5378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:49.039314 5378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:49.039443 5378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:49.039569 5378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:49.039655 5378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:49.071122 5378 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38711
I20260927 06:54:49.071174 5504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38711 every 8 connection(s)
I20260927 06:54:49.072445 5378 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:54:49.073369 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 5378
I20260927 06:54:49.073575 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:54:49.074043 388 external_mini_cluster.cc:1428] Setting key 26c3b88ac1446ce6afee0cd77f3bc350
I20260927 06:54:49.080602 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.075090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43303 (local address 127.0.97.62:33573)
0927 06:54:49.075294 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:49.075306 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:49.077187 (+ 1881us) server_negotiation.cc:407] Connection header received
0927 06:54:49.078692 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:49.078698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:49.078759 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:49.078852 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:49.080068 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:49.080072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:49.080075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:49.080091 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:49.080122 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:49.080134 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:49.080229 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:49.080358 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 06:54:49.080404 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 06:54:49.081434 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.075739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33573 (local address 127.0.97.3:43303)
0927 06:54:49.076801 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:49.076888 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:49.078459 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:49.079074 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:49.079081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:49.079163 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:49.079739 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:54:49.079742 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:49.079781 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:54:49.079903 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:49.080202 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:49.080207 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:49.080211 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 06:54:49.080400 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 06:54:49.080568 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":323,"threads_started":1}
I20260927 06:54:49.082762 5505 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33573
I20260927 06:54:49.083042 5505 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:49.083631 5505 heartbeater.cc:511] Master 127.0.97.62:33573 requested a full tablet report, sending...
I20260927 06:54:49.084926 5059 ts_manager.cc:194] Registered new tserver with Master: e3ed8678c7cf4d14ae8cef29c8577cbd (127.0.97.3:38711)
I20260927 06:54:49.089165 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:54:49.095621 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.093420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.0.97.62:33573)
0927 06:54:49.093587 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:49.093591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:49.094125 (+ 534us) server_negotiation.cc:407] Connection header received
0927 06:54:49.094243 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:49.094246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:49.094288 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:49.094361 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:49.095153 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:49.095157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:49.095160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:49.095174 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:49.095201 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:49.095212 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:49.095300 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:49.095370 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 06:54:49.095409 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 06:54:49.099192 5059 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:54:49.105357 5059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:54:49.121917 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.116441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46163 (local address 127.0.0.1:46734)
0927 06:54:49.118264 (+ 1823us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:49.118296 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:49.118433 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:49.119007 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:49.119014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:49.119096 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:49.119954 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:54:49.119957 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:49.119983 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:54:49.120063 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:49.121076 (+ 1013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:49.121092 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:49.121105 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 06:54:49.121320 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 06:54:49.121411 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1589,"thread_start_us":1423,"threads_started":1}
I20260927 06:54:49.123431 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.116816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.0.97.1:46163)
0927 06:54:49.117678 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:49.117691 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:49.118470 (+ 779us) server_negotiation.cc:407] Connection header received
0927 06:54:49.118547 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:49.118557 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:49.118793 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:49.119533 (+ 740us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:49.120838 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:49.120847 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:49.120865 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:49.120892 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:49.120951 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:49.120971 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:49.122895 (+ 1924us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:49.122966 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 06:54:49.123064 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":552,"threads_started":1}
I20260927 06:54:49.124507 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.116652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38711 (local address 127.0.0.1:54806)
0927 06:54:49.118928 (+ 2276us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:49.118945 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:49.119019 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:49.120984 (+ 1965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:49.120991 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:49.121018 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:49.121771 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:54:49.121779 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:49.121795 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:54:49.121841 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:49.123941 (+ 2100us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:49.123946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:49.123951 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 06:54:49.124065 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 06:54:49.124110 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2128,"thread_start_us":1994,"threads_started":1}
I20260927 06:54:49.125248 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.117073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.0.97.3:38711)
0927 06:54:49.120515 (+ 3442us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:49.120522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:49.120551 (+ 29us) server_negotiation.cc:407] Connection header received
0927 06:54:49.120609 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:49.120616 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:49.120812 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:49.121454 (+ 642us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:49.123698 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:49.123706 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:49.123719 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:49.123738 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:49.123795 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:49.123814 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:49.124747 (+ 933us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:49.124807 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 06:54:49.124896 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3272,"thread_start_us":2158,"threads_started":1}
I20260927 06:54:49.127338 5180 tablet_service.cc:1467] Processing CreateTablet for tablet 502828eabd764ab9a06d4026a0dd583a (DEFAULT_TABLE table=test-table [id=a5a4bd33b305446e89b091f52584eb45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:49.128746 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.119087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34599 (local address 127.0.0.1:34688)
0927 06:54:49.122627 (+ 3540us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:49.122648 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:49.122719 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:49.126769 (+ 4050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:49.126774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:49.126806 (+ 32us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:49.127789 (+ 983us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:54:49.127792 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:49.127809 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:54:49.127850 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:49.128330 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:49.128336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:49.128343 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 06:54:49.128437 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 06:54:49.128496 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3441,"thread_start_us":7192,"threads_started":1}
I20260927 06:54:49.128791 5180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502828eabd764ab9a06d4026a0dd583a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:49.129074 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.120140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.0.97.2:34599)
0927 06:54:49.126255 (+ 6115us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:49.126263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:49.126289 (+ 26us) server_negotiation.cc:407] Connection header received
0927 06:54:49.126360 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:49.126367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:49.126548 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:49.127156 (+ 608us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:49.128092 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:49.128099 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:49.128111 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:49.128136 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:49.128197 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:49.128215 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:49.128383 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:49.128451 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 06:54:49.128525 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5960,"thread_start_us":5818,"threads_started":1}
I20260927 06:54:49.129922 5440 tablet_service.cc:1467] Processing CreateTablet for tablet 502828eabd764ab9a06d4026a0dd583a (DEFAULT_TABLE table=test-table [id=a5a4bd33b305446e89b091f52584eb45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:49.131300 5440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502828eabd764ab9a06d4026a0dd583a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:49.132319 5310 tablet_service.cc:1467] Processing CreateTablet for tablet 502828eabd764ab9a06d4026a0dd583a (DEFAULT_TABLE table=test-table [id=a5a4bd33b305446e89b091f52584eb45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:54:49.133556 5310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502828eabd764ab9a06d4026a0dd583a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:49.140164 5523 tablet_bootstrap.cc:492] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3: Bootstrap starting.
I20260927 06:54:49.143006 5523 tablet_bootstrap.cc:654] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:49.143935 5523 log.cc:824] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:49.146855 5524 tablet_bootstrap.cc:492] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd: Bootstrap starting.
I20260927 06:54:49.148062 5526 tablet_bootstrap.cc:492] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b: Bootstrap starting.
I20260927 06:54:49.149351 5523 tablet_bootstrap.cc:492] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3: No bootstrap required, opened a new log
I20260927 06:54:49.149607 5523 ts_tablet_manager.cc:1397] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260927 06:54:49.151255 5526 tablet_bootstrap.cc:654] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:49.151271 5524 tablet_bootstrap.cc:654] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:49.152446 5526 log.cc:824] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:49.152446 5524 log.cc:824] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:49.156392 5526 tablet_bootstrap.cc:492] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b: No bootstrap required, opened a new log
I20260927 06:54:49.156649 5526 ts_tablet_manager.cc:1397] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 06:54:49.157239 5524 tablet_bootstrap.cc:492] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd: No bootstrap required, opened a new log
I20260927 06:54:49.157478 5524 ts_tablet_manager.cc:1397] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260927 06:54:49.157677 5523 raft_consensus.cc:348] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee037ebd5bbf44bfa4c0191ddbec7bf3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46163 } } peers { permanent_uuid: "e3ed8678c7cf4d14ae8cef29c8577cbd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } } peers { permanent_uuid: "65ed0a2b135846dc8b70691a4441c98b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34599 } }
I20260927 06:54:49.158075 5523 raft_consensus.cc:374] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:49.158188 5523 raft_consensus.cc:729] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee037ebd5bbf44bfa4c0191ddbec7bf3, State: Initialized, Role: FOLLOWER
I20260927 06:54:49.158774 5523 consensus_queue.cc:260] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee037ebd5bbf44bfa4c0191ddbec7bf3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46163 } } peers { permanent_uuid: "e3ed8678c7cf4d14ae8cef29c8577cbd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } } peers { permanent_uuid: "65ed0a2b135846dc8b70691a4441c98b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34599 } }
I20260927 06:54:49.159825 5245 heartbeater.cc:503] Master 127.0.97.62:33573 was elected leader, sending a full tablet report...
I20260927 06:54:49.160444 5523 ts_tablet_manager.cc:1428] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260927 06:54:49.166934 5526 raft_consensus.cc:348] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee037ebd5bbf44bfa4c0191ddbec7bf3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46163 } } peers { permanent_uuid: "e3ed8678c7cf4d14ae8cef29c8577cbd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } } peers { permanent_uuid: "65ed0a2b135846dc8b70691a4441c98b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34599 } }
I20260927 06:54:49.167322 5526 raft_consensus.cc:374] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:49.167436 5526 raft_consensus.cc:729] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ed0a2b135846dc8b70691a4441c98b, State: Initialized, Role: FOLLOWER
I20260927 06:54:49.167588 5524 raft_consensus.cc:348] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee037ebd5bbf44bfa4c0191ddbec7bf3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46163 } } peers { permanent_uuid: "e3ed8678c7cf4d14ae8cef29c8577cbd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } } peers { permanent_uuid: "65ed0a2b135846dc8b70691a4441c98b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34599 } }
I20260927 06:54:49.167976 5524 raft_consensus.cc:374] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:49.168094 5524 raft_consensus.cc:729] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ed8678c7cf4d14ae8cef29c8577cbd, State: Initialized, Role: FOLLOWER
I20260927 06:54:49.168026 5526 consensus_queue.cc:260] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee037ebd5bbf44bfa4c0191ddbec7bf3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46163 } } peers { permanent_uuid: "e3ed8678c7cf4d14ae8cef29c8577cbd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } } peers { permanent_uuid: "65ed0a2b135846dc8b70691a4441c98b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34599 } }
I20260927 06:54:49.168638 5524 consensus_queue.cc:260] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee037ebd5bbf44bfa4c0191ddbec7bf3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46163 } } peers { permanent_uuid: "e3ed8678c7cf4d14ae8cef29c8577cbd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } } peers { permanent_uuid: "65ed0a2b135846dc8b70691a4441c98b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34599 } }
I20260927 06:54:49.168969 5375 heartbeater.cc:503] Master 127.0.97.62:33573 was elected leader, sending a full tablet report...
I20260927 06:54:49.169729 5526 ts_tablet_manager.cc:1428] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260927 06:54:49.170478 5505 heartbeater.cc:503] Master 127.0.97.62:33573 was elected leader, sending a full tablet report...
I20260927 06:54:49.171092 5524 ts_tablet_manager.cc:1428] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
W20260927 06:54:49.248824 5246 tablet.cc:2369] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:54:49.324056 5506 tablet.cc:2369] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:54:49.403607 5529 raft_consensus.cc:482] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:54:49.403892 5529 raft_consensus.cc:504] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee037ebd5bbf44bfa4c0191ddbec7bf3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46163 } } peers { permanent_uuid: "e3ed8678c7cf4d14ae8cef29c8577cbd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } } peers { permanent_uuid: "65ed0a2b135846dc8b70691a4441c98b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34599 } }
I20260927 06:54:49.405603 5529 leader_election.cc:288] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3ed8678c7cf4d14ae8cef29c8577cbd (127.0.97.3:38711), 65ed0a2b135846dc8b70691a4441c98b (127.0.97.2:34599)
I20260927 06:54:49.412416 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.406933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59611 (local address 127.0.97.2:34599)
0927 06:54:49.407103 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:49.407112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:49.407474 (+ 362us) server_negotiation.cc:407] Connection header received
0927 06:54:49.407687 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:49.407693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:49.407755 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:49.407861 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:49.411748 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:49.411755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:49.411759 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:49.411780 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:49.411818 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:49.411829 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:49.411923 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:49.412070 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 06:54:49.412136 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 06:54:49.412447 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.405957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55057 (local address 127.0.97.3:38711)
0927 06:54:49.406174 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:49.406179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:49.406819 (+ 640us) server_negotiation.cc:407] Connection header received
0927 06:54:49.407216 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:49.407221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:49.407281 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:49.407350 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:49.411799 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:49.411803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:49.411806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:49.411821 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:49.411847 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:49.411858 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:49.412016 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:49.412135 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 06:54:49.412189 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260927 06:54:49.412463 5376 tablet.cc:2369] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:54:49.412878 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.405954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38711 (local address 127.0.97.1:55057)
0927 06:54:49.406717 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:49.406736 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:49.406909 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:49.407550 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:49.407553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:49.407571 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:49.411606 (+ 4035us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:54:49.411609 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:49.411621 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:54:49.411655 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:49.411991 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:49.411995 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:49.412016 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 06:54:49.412174 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 06:54:49.412219 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":410,"threads_started":1}
I20260927 06:54:49.413465 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.406796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34599 (local address 127.0.97.1:59611)
0927 06:54:49.407374 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:49.407395 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:49.407556 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:49.407880 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:49.407883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:49.407899 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:49.411553 (+ 3654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:54:49.411559 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:49.411574 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:54:49.411611 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:49.411931 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:49.411937 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:49.411945 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 06:54:49.412094 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 06:54:49.412147 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":295,"threads_started":1}
I20260927 06:54:49.413899 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502828eabd764ab9a06d4026a0dd583a" candidate_uuid: "ee037ebd5bbf44bfa4c0191ddbec7bf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3ed8678c7cf4d14ae8cef29c8577cbd" is_pre_election: true
I20260927 06:54:49.414340 5330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502828eabd764ab9a06d4026a0dd583a" candidate_uuid: "ee037ebd5bbf44bfa4c0191ddbec7bf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65ed0a2b135846dc8b70691a4441c98b" is_pre_election: true
I20260927 06:54:49.414503 5460 raft_consensus.cc:2454] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee037ebd5bbf44bfa4c0191ddbec7bf3 in term 0.
I20260927 06:54:49.414803 5330 raft_consensus.cc:2454] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee037ebd5bbf44bfa4c0191ddbec7bf3 in term 0.
I20260927 06:54:49.415218 5134 leader_election.cc:302] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3ed8678c7cf4d14ae8cef29c8577cbd, ee037ebd5bbf44bfa4c0191ddbec7bf3; no voters:
I20260927 06:54:49.415570 5529 raft_consensus.cc:2781] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:54:49.415714 5529 raft_consensus.cc:482] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:54:49.415838 5529 raft_consensus.cc:3037] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:49.417950 5529 raft_consensus.cc:504] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee037ebd5bbf44bfa4c0191ddbec7bf3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46163 } } peers { permanent_uuid: "e3ed8678c7cf4d14ae8cef29c8577cbd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } } peers { permanent_uuid: "65ed0a2b135846dc8b70691a4441c98b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34599 } }
I20260927 06:54:49.418560 5529 leader_election.cc:288] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [CANDIDATE]: Term 1 election: Requested vote from peers e3ed8678c7cf4d14ae8cef29c8577cbd (127.0.97.3:38711), 65ed0a2b135846dc8b70691a4441c98b (127.0.97.2:34599)
I20260927 06:54:49.418983 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502828eabd764ab9a06d4026a0dd583a" candidate_uuid: "ee037ebd5bbf44bfa4c0191ddbec7bf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3ed8678c7cf4d14ae8cef29c8577cbd"
I20260927 06:54:49.419112 5330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502828eabd764ab9a06d4026a0dd583a" candidate_uuid: "ee037ebd5bbf44bfa4c0191ddbec7bf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65ed0a2b135846dc8b70691a4441c98b"
I20260927 06:54:49.419168 5460 raft_consensus.cc:3037] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:49.419317 5330 raft_consensus.cc:3037] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:49.421231 5460 raft_consensus.cc:2454] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee037ebd5bbf44bfa4c0191ddbec7bf3 in term 1.
I20260927 06:54:49.421423 5330 raft_consensus.cc:2454] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee037ebd5bbf44bfa4c0191ddbec7bf3 in term 1.
I20260927 06:54:49.421614 5134 leader_election.cc:302] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3ed8678c7cf4d14ae8cef29c8577cbd, ee037ebd5bbf44bfa4c0191ddbec7bf3; no voters:
I20260927 06:54:49.421883 5529 raft_consensus.cc:2781] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:49.422403 5529 raft_consensus.cc:686] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [term 1 LEADER]: Becoming Leader. State: Replica: ee037ebd5bbf44bfa4c0191ddbec7bf3, State: Running, Role: LEADER
I20260927 06:54:49.422866 5529 consensus_queue.cc:237] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee037ebd5bbf44bfa4c0191ddbec7bf3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46163 } } peers { permanent_uuid: "e3ed8678c7cf4d14ae8cef29c8577cbd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } } peers { permanent_uuid: "65ed0a2b135846dc8b70691a4441c98b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34599 } }
I20260927 06:54:49.427246 5058 catalog_manager.cc:5495] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee037ebd5bbf44bfa4c0191ddbec7bf3 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ee037ebd5bbf44bfa4c0191ddbec7bf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee037ebd5bbf44bfa4c0191ddbec7bf3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3ed8678c7cf4d14ae8cef29c8577cbd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65ed0a2b135846dc8b70691a4441c98b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34599 } health_report { overall_health: UNKNOWN } } }
I20260927 06:54:49.486835 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.484874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53194 (local address 127.0.97.1:46163)
0927 06:54:49.485094 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:49.485099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:49.485117 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:54:49.485280 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:49.485283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:49.485358 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:49.485439 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:49.486239 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:49.486243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:49.486246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:49.486261 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:49.486291 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:49.486301 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:49.486393 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:49.486460 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 06:54:49.486502 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 06:54:49.494674 5460 raft_consensus.cc:1261] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd [term 1 FOLLOWER]: Refusing update from remote peer ee037ebd5bbf44bfa4c0191ddbec7bf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:54:49.494699 5330 raft_consensus.cc:1261] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b [term 1 FOLLOWER]: Refusing update from remote peer ee037ebd5bbf44bfa4c0191ddbec7bf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:54:49.495513 5529 consensus_queue.cc:1035] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3ed8678c7cf4d14ae8cef29c8577cbd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:49.495832 5534 consensus_queue.cc:1035] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65ed0a2b135846dc8b70691a4441c98b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:54:49.507437 5539 mvcc.cc:205] Tried to move back new op lower bound from 7333855598561083392 to 7333855598279454720. Current Snapshot: MvccSnapshot[applied={T|T < 7333855598561083392}]
I20260927 06:54:49.518443 5059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:54:49.518642 5059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:54:49.521924 5059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b18c6a829e140a381d8c37e256e82cb: Sending DeleteTablet for 3 replicas of tablet 502828eabd764ab9a06d4026a0dd583a
I20260927 06:54:49.522960 5180 tablet_service.cc:1514] Processing DeleteTablet for tablet 502828eabd764ab9a06d4026a0dd583a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:49 UTC) from {username='slave'} at 127.0.0.1:46734
I20260927 06:54:49.523061 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet 502828eabd764ab9a06d4026a0dd583a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:49 UTC) from {username='slave'} at 127.0.0.1:54806
I20260927 06:54:49.523304 5310 tablet_service.cc:1514] Processing DeleteTablet for tablet 502828eabd764ab9a06d4026a0dd583a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:54:49 UTC) from {username='slave'} at 127.0.0.1:34688
I20260927 06:54:49.524712 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 5118
I20260927 06:54:49.547847 5548 tablet_replica.cc:330] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b: stopping tablet replica
I20260927 06:54:49.548152 5547 tablet_replica.cc:330] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd: stopping tablet replica
I20260927 06:54:49.548655 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.525480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53196 (local address 127.0.97.1:46163)
0927 06:54:49.525636 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:49.525642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:49.546051 (+ 20409us) server_negotiation.cc:407] Connection header received
0927 06:54:49.546526 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:49.546532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:49.546596 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:49.546708 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:49.547864 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:49.547869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:49.547874 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:49.547893 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:49.547931 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:49.547946 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:49.548045 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:49.548285 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 06:54:49.548361 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 06:54:49.549293 5547 raft_consensus.cc:2229] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:54:49.549727 5546 tablet_replica.cc:330] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3: stopping tablet replica
I20260927 06:54:49.549786 5240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:49.549819 5547 raft_consensus.cc:2258] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:49.550503 5548 raft_consensus.cc:2229] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:54:49.550979 5548 raft_consensus.cc:2258] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:54:49.552639 5547 ts_tablet_manager.cc:1905] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:49.553943 5548 ts_tablet_manager.cc:1905] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:54:49.559492 5547 ts_tablet_manager.cc:1918] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:54:49.559641 5547 log.cc:1196] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/502828eabd764ab9a06d4026a0dd583a
I20260927 06:54:49.560113 5547 ts_tablet_manager.cc:1939] T 502828eabd764ab9a06d4026a0dd583a P e3ed8678c7cf4d14ae8cef29c8577cbd: Deleting consensus metadata
I20260927 06:54:49.560612 5548 ts_tablet_manager.cc:1918] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:54:49.560767 5548 log.cc:1196] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/502828eabd764ab9a06d4026a0dd583a
I20260927 06:54:49.561240 5548 ts_tablet_manager.cc:1939] T 502828eabd764ab9a06d4026a0dd583a P 65ed0a2b135846dc8b70691a4441c98b: Deleting consensus metadata
I20260927 06:54:49.562005 5044 catalog_manager.cc:4841] TS e3ed8678c7cf4d14ae8cef29c8577cbd (127.0.97.3:38711): tablet 502828eabd764ab9a06d4026a0dd583a (table test-table [id=a5a4bd33b305446e89b091f52584eb45]) successfully deleted
I20260927 06:54:49.562788 5044 catalog_manager.cc:4841] TS 65ed0a2b135846dc8b70691a4441c98b (127.0.97.2:34599): tablet 502828eabd764ab9a06d4026a0dd583a (table test-table [id=a5a4bd33b305446e89b091f52584eb45]) successfully deleted
I20260927 06:54:49.550252 5546 raft_consensus.cc:2229] T 502828eabd764ab9a06d4026a0dd583a P ee037ebd5bbf44bfa4c0191ddbec7bf3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:54:49.650705 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 5118
W20260927 06:54:49.671448 5046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:46163 (error 108)
I20260927 06:54:49.671588 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 5248
I20260927 06:54:49.674845 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.672947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46163 (local address 127.0.0.1:53212)
0927 06:54:49.673884 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:49.674305 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":697,"threads_started":1}
I20260927 06:54:49.674968 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.672631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.0.97.2:34599)
0927 06:54:49.672820 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:49.672828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:49.672852 (+ 24us) server_negotiation.cc:407] Connection header received
0927 06:54:49.672924 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:49.672928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:49.673039 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:49.673167 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:49.674242 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:49.674250 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:49.674257 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:49.674280 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:49.674324 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:49.674344 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:49.674507 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:49.674560 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 06:54:49.674625 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 06:54:49.675035 5046 catalog_manager.cc:4584] TS ee037ebd5bbf44bfa4c0191ddbec7bf3 (127.0.97.1:46163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 502828eabd764ab9a06d4026a0dd583a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46163: connect: Connection refused (error 111)
I20260927 06:54:49.675709 5370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:49.728490 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.728059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46163 (local address 127.0.0.1:53218)
0927 06:54:49.728260 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:49.728359 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 06:54:49.729417 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.729100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46163 (local address 127.0.0.1:53234)
0927 06:54:49.729221 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:49.729281 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 06:54:49.763957 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 5248
I20260927 06:54:49.780604 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 5378
I20260927 06:54:49.783298 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.781347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42234 (local address 127.0.97.3:38711)
0927 06:54:49.781562 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:49.781569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:49.781590 (+ 21us) server_negotiation.cc:407] Connection header received
0927 06:54:49.781744 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:49.781748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:49.781831 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:49.781913 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:49.782739 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:49.782743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:49.782746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:49.782760 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:49.782787 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:49.782798 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:49.782895 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:49.782965 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 06:54:49.783005 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 06:54:49.783950 5500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:49.789582 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.789213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46163 (local address 127.0.0.1:53238)
0927 06:54:49.789366 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:49.789436 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 06:54:49.790489 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.790154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46163 (local address 127.0.0.1:53252)
0927 06:54:49.790290 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:49.790365 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 06:54:49.867702 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 5378
I20260927 06:54:49.871109 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.870683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46163 (local address 127.0.0.1:53254)
0927 06:54:49.870886 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:49.870973 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 06:54:49.872164 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:49.871769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46163 (local address 127.0.0.1:53260)
0927 06:54:49.871957 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:49.872039 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 06:54:49.884454 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 5028
I20260927 06:54:49.885496 5089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:49.985996 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 5028
2026-09-27T06:54:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2684 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 06:54:50.026414 388 mini_oidc.cc:152] Starting JWKS server
I20260927 06:54:50.028888 388 webserver.cc:460] Webserver started at https://127.0.0.1:40303/ using document root <none> and password file <none>
I20260927 06:54:50.029227 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 06:54:50.029850 388 webserver.cc:460] Webserver started at http://127.0.0.1:40371/ using document root <none> and password file <none>
2026-09-27T06:54:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:54:50Z Disabled control of system clock
I20260927 06:54:50.055354 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45385
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45385
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40303/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/testchainca.cert with env {}
W20260927 06:54:50.365094 5561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:50.365370 5561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:50.365454 5561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:50.372617 5561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:54:50.372709 5561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:50.372745 5561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:50.372807 5561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:54:50.372849 5561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 06:54:50.372900 5561 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 06:54:50.372937 5561 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40303/jwks.json
I20260927 06:54:50.382563 5561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45385
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40303/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:50.383769 5561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:50.384996 5561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:50.396407 5569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:50.396521 5566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:50.396500 5567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:50.397315 5561 server_base.cc:1034] running on GCE node
I20260927 06:54:50.398150 5561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:50.399490 5561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:50.400710 5561 hybrid_clock.cc:648] HybridClock initialized: now 1790492090400675 us; error 48 us; skew 500 ppm
I20260927 06:54:50.401049 5561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:54:50.404223 5561 webserver.cc:460] Webserver started at http://127.0.97.62:46827/ using document root <none> and password file <none>
I20260927 06:54:50.404882 5561 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:50.405005 5561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:50.405301 5561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:50.408164 5561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "8a090b3054fe4e5eb00822586c64fbf0"
format_stamp: "Formatted at 2026-09-27 06:54:50 on dist-test-slave-f892"
server_key: "0426337ff2cc005b4ce84f0ff1147ad7"
server_key_iv: "813b0a77e82de3350b56f56ab509479f"
server_key_version: "encryptionkey@0"
I20260927 06:54:50.408934 5561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "8a090b3054fe4e5eb00822586c64fbf0"
format_stamp: "Formatted at 2026-09-27 06:54:50 on dist-test-slave-f892"
server_key: "0426337ff2cc005b4ce84f0ff1147ad7"
server_key_iv: "813b0a77e82de3350b56f56ab509479f"
server_key_version: "encryptionkey@0"
I20260927 06:54:50.413628 5561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260927 06:54:50.416781 5575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:50.418084 5561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 06:54:50.418256 5561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "8a090b3054fe4e5eb00822586c64fbf0"
format_stamp: "Formatted at 2026-09-27 06:54:50 on dist-test-slave-f892"
server_key: "0426337ff2cc005b4ce84f0ff1147ad7"
server_key_iv: "813b0a77e82de3350b56f56ab509479f"
server_key_version: "encryptionkey@0"
I20260927 06:54:50.418470 5561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:50.444372 5561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:50.445156 5561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:50.445438 5561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:50.463028 5561 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45385
I20260927 06:54:50.463061 5626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45385 every 8 connection(s)
I20260927 06:54:50.464268 5561 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:54:50.467700 5627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:50.471352 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 5561
I20260927 06:54:50.471549 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:54:50.472306 388 external_mini_cluster.cc:1428] Setting key 2e0c1955d8e62a7166c26525db3e50fd
I20260927 06:54:50.479755 5627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0: Bootstrap starting.
I20260927 06:54:50.482705 5627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:50.483861 5627 log.cc:824] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:50.484899 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:50.474178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34202 (local address 127.0.97.62:45385)
0927 06:54:50.475041 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:50.475051 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:50.475090 (+ 39us) server_negotiation.cc:407] Connection header received
0927 06:54:50.476622 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:50.476643 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:50.477088 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:50.477479 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:50.478085 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:50.479220 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:50.480523 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:50.481151 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:50.481365 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:50.482593 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:50.482601 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:50.482613 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:50.483150 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:50.483250 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:50.483295 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:50.483437 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:50.483680 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 06:54:50.483949 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":297,"threads_started":1}
I20260927 06:54:50.486814 5627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0: No bootstrap required, opened a new log
I20260927 06:54:50.488587 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:54:50.494971 5627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a090b3054fe4e5eb00822586c64fbf0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45385 } }
I20260927 06:54:50.495273 5627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:50.495342 5627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a090b3054fe4e5eb00822586c64fbf0, State: Initialized, Role: FOLLOWER
I20260927 06:54:50.495774 5627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a090b3054fe4e5eb00822586c64fbf0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45385 } }
I20260927 06:54:50.495976 5627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:54:50.496078 5627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:54:50.496181 5627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:50.498324 5627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a090b3054fe4e5eb00822586c64fbf0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45385 } }
I20260927 06:54:50.498801 5627 leader_election.cc:302] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a090b3054fe4e5eb00822586c64fbf0; no voters:
I20260927 06:54:50.499280 5627 leader_election.cc:288] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:54:50.499432 5632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:50.500043 5632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0 [term 1 LEADER]: Becoming Leader. State: Replica: 8a090b3054fe4e5eb00822586c64fbf0, State: Running, Role: LEADER
I20260927 06:54:50.500530 5632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a090b3054fe4e5eb00822586c64fbf0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45385 } }
I20260927 06:54:50.501190 5627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:54:50.503432 5634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a090b3054fe4e5eb00822586c64fbf0. Latest consensus state: current_term: 1 leader_uuid: "8a090b3054fe4e5eb00822586c64fbf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a090b3054fe4e5eb00822586c64fbf0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45385 } } }
I20260927 06:54:50.503563 5633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a090b3054fe4e5eb00822586c64fbf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a090b3054fe4e5eb00822586c64fbf0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45385 } } }
I20260927 06:54:50.503875 5633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:50.503875 5634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:50.505870 5637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:54:50.511088 5637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:54:50.519731 5637 catalog_manager.cc:1347] Generated new cluster ID: b3817acbe04b4e5fa2a11cde7caca8ec
I20260927 06:54:50.519835 5637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:54:50.534216 5637 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:54:50.535343 5637 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:54:50.545110 5637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a090b3054fe4e5eb00822586c64fbf0: Generated new TSK 0
I20260927 06:54:50.545997 5637 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 06:54:50.598663 5657 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 06:54:50.598819 5657 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 06:54:50.599599 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:50.571108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34210 (local address 127.0.97.62:45385)
0927 06:54:50.571330 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:50.571339 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:50.571877 (+ 538us) server_negotiation.cc:407] Connection header received
0927 06:54:50.572184 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:50.572188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:50.572249 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:50.572374 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0927 06:54:50.572980 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:50.573666 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:50.575661 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:50.576070 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:50.576164 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:50.576738 (+ 574us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 06:54:50.599082 (+ 22344us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40303/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 06:54:50.599375 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34210: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40303/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 06:54:50.599737 5630 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34210: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40303/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 06:54:50.601428 5656 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:54:50.571116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45385 (local address 127.0.0.1:34210)
0927 06:54:50.571768 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:50.571787 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:50.571962 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:50.572430 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:50.572435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:50.572448 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0927 06:54:50.572836 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:50.572847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:50.573786 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:50.573792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:50.575534 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:50.575548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:50.576181 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:50.576239 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:50.576418 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:50.576498 (+ 80us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 06:54:50.599351 (+ 22853us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40303/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 06:54:50.599699 (+ 348us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:45385: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40303/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":337,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:45385: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40303/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 06:54:50.603644 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 5561
I20260927 06:54:50.604739 5622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:50.695197 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 5561
2026-09-27T06:54:50Z chronyd exiting
I20260927 06:54:50.744737 388 test_util.cc:175] -----------------------------------------------
I20260927 06:54:50.744858 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492047466241-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (722 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 06:54:50.748481 388 mini_oidc.cc:152] Starting JWKS server
I20260927 06:54:50.750207 388 webserver.cc:460] Webserver started at https://127.0.0.1:35411/ using document root <none> and password file <none>
I20260927 06:54:50.750425 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 06:54:50.751130 388 webserver.cc:460] Webserver started at http://127.0.0.1:44479/ using document root <none> and password file <none>
2026-09-27T06:54:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:54:50Z Disabled control of system clock
I20260927 06:54:50.774173 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43429
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43429
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35411/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/testchainca.cert with env {}
W20260927 06:54:51.077031 5664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:51.077319 5664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:51.077404 5664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:51.084417 5664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:54:51.084522 5664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:51.084611 5664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:51.084769 5664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:54:51.084822 5664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 06:54:51.084924 5664 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 06:54:51.084980 5664 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35411/jwks.json
I20260927 06:54:51.094645 5664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43429
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35411/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:51.095852 5664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:51.097118 5664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:51.108201 5669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:51.108237 5672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:51.108451 5670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:51.109064 5664 server_base.cc:1034] running on GCE node
I20260927 06:54:51.109743 5664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:51.111063 5664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:51.112315 5664 hybrid_clock.cc:648] HybridClock initialized: now 1790492091112270 us; error 49 us; skew 500 ppm
I20260927 06:54:51.112650 5664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:54:51.114873 5664 webserver.cc:460] Webserver started at http://127.0.97.62:46411/ using document root <none> and password file <none>
I20260927 06:54:51.115530 5664 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:51.115655 5664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:51.115959 5664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:51.118727 5664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "6a302402b6054b02a18048d57cbe44e4"
format_stamp: "Formatted at 2026-09-27 06:54:51 on dist-test-slave-f892"
server_key: "63a81b3b3610061376fb98ff8c5ada21"
server_key_iv: "387c4192ff2282f116a058f002314034"
server_key_version: "encryptionkey@0"
I20260927 06:54:51.119516 5664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "6a302402b6054b02a18048d57cbe44e4"
format_stamp: "Formatted at 2026-09-27 06:54:51 on dist-test-slave-f892"
server_key: "63a81b3b3610061376fb98ff8c5ada21"
server_key_iv: "387c4192ff2282f116a058f002314034"
server_key_version: "encryptionkey@0"
I20260927 06:54:51.124148 5664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 06:54:51.127209 5678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:51.128396 5664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 06:54:51.128568 5664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "6a302402b6054b02a18048d57cbe44e4"
format_stamp: "Formatted at 2026-09-27 06:54:51 on dist-test-slave-f892"
server_key: "63a81b3b3610061376fb98ff8c5ada21"
server_key_iv: "387c4192ff2282f116a058f002314034"
server_key_version: "encryptionkey@0"
I20260927 06:54:51.128798 5664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:51.142586 5664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:51.143268 5664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:51.143548 5664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:51.161180 5664 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43429
I20260927 06:54:51.161187 5729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43429 every 8 connection(s)
I20260927 06:54:51.162732 5664 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:54:51.166291 5730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:51.169238 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 5664
I20260927 06:54:51.169441 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:54:51.170215 388 external_mini_cluster.cc:1428] Setting key 498231111c3a2c395cd1b2d5a670f00b
I20260927 06:54:51.178210 5730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4: Bootstrap starting.
I20260927 06:54:51.180941 5730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:51.181941 5730 log.cc:824] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:51.183079 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:51.172081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.0.97.62:43429)
0927 06:54:51.172988 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:51.172997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:51.173035 (+ 38us) server_negotiation.cc:407] Connection header received
0927 06:54:51.175103 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:51.175123 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:51.175530 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:51.175912 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:51.176446 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:51.177628 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:51.178934 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:51.179584 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:51.179781 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:51.180843 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:51.180850 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:51.180866 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:51.181418 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:51.181497 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:51.181521 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:51.181645 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:51.181844 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 06:54:51.182190 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":338,"threads_started":1}
I20260927 06:54:51.184504 5730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4: No bootstrap required, opened a new log
I20260927 06:54:51.186465 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:54:51.192145 5730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a302402b6054b02a18048d57cbe44e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43429 } }
I20260927 06:54:51.192543 5730 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:51.192679 5730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a302402b6054b02a18048d57cbe44e4, State: Initialized, Role: FOLLOWER
I20260927 06:54:51.193207 5730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a302402b6054b02a18048d57cbe44e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43429 } }
I20260927 06:54:51.193441 5730 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:54:51.193583 5730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:54:51.193740 5730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:51.195878 5730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a302402b6054b02a18048d57cbe44e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43429 } }
I20260927 06:54:51.196451 5730 leader_election.cc:302] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a302402b6054b02a18048d57cbe44e4; no voters:
I20260927 06:54:51.196988 5730 leader_election.cc:288] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:54:51.197135 5735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:51.197839 5735 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4 [term 1 LEADER]: Becoming Leader. State: Replica: 6a302402b6054b02a18048d57cbe44e4, State: Running, Role: LEADER
I20260927 06:54:51.198383 5735 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a302402b6054b02a18048d57cbe44e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43429 } }
I20260927 06:54:51.199082 5730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:54:51.200884 5736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a302402b6054b02a18048d57cbe44e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a302402b6054b02a18048d57cbe44e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43429 } } }
I20260927 06:54:51.201175 5736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:51.201637 5737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a302402b6054b02a18048d57cbe44e4. Latest consensus state: current_term: 1 leader_uuid: "6a302402b6054b02a18048d57cbe44e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a302402b6054b02a18048d57cbe44e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43429 } } }
I20260927 06:54:51.201959 5737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:51.202878 5741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:54:51.207971 5741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:54:51.216707 5741 catalog_manager.cc:1347] Generated new cluster ID: aa32915835fa4a919fdeb4b9959ef6ff
I20260927 06:54:51.216809 5741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:54:51.225904 5741 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:54:51.227113 5741 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:54:51.237150 5741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a302402b6054b02a18048d57cbe44e4: Generated new TSK 0
I20260927 06:54:51.237943 5741 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 06:54:51.267534 5760 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 06:54:51.267656 5760 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260927 06:54:51.268743 5759 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:54:51.249431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43429 (local address 127.0.0.1:48346)
0927 06:54:51.250092 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:51.250116 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:51.250220 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:51.250701 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:51.250704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:51.250712 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 06:54:51.250958 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:51.250966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:51.252328 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:51.252332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:51.253360 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:51.253371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:51.254318 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:51.254353 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:51.254483 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:51.254512 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 06:54:51.268318 (+ 13806us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 06:54:51.268484 (+ 166us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:43429: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":345,"threads_started":1}
I20260927 06:54:51.268750 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:51.249554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.0.97.62:43429)
0927 06:54:51.249713 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:51.249719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:51.250189 (+ 470us) server_negotiation.cc:407] Connection header received
0927 06:54:51.250382 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:51.250391 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:51.250511 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:51.250679 (+ 168us) server_negotiation.cc:226] Negotiated authn=JWT
0927 06:54:51.251094 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:51.252194 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:51.253498 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:51.254214 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:51.254333 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:51.254746 (+ 413us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 06:54:51.268129 (+ 13383us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 06:54:51.268484 (+ 355us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48346: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 06:54:51.268857 5733 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48346: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 06:54:51.269730 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 5664
I20260927 06:54:51.272148 5725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:51.363193 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 5664
2026-09-27T06:54:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (667 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 06:54:51.416062 388 mini_oidc.cc:152] Starting JWKS server
I20260927 06:54:51.417824 388 webserver.cc:460] Webserver started at https://127.0.0.1:45285/ using document root <none> and password file <none>
I20260927 06:54:51.418124 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 06:54:51.418721 388 webserver.cc:460] Webserver started at http://127.0.0.1:38405/ using document root <none> and password file <none>
2026-09-27T06:54:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:54:51Z Disabled control of system clock
I20260927 06:54:51.444227 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36813
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36813
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45285/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 06:54:51.751485 5767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:51.751754 5767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:51.751940 5767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:51.758944 5767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:54:51.759081 5767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:51.759150 5767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:51.759253 5767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:54:51.759390 5767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 06:54:51.759483 5767 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 06:54:51.759570 5767 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45285/jwks.json
I20260927 06:54:51.768288 5767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45285/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:51.769548 5767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:51.771318 5767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:51.782703 5772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:51.782652 5773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:51.782740 5775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:51.783497 5767 server_base.cc:1034] running on GCE node
I20260927 06:54:51.784133 5767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:51.785326 5767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:51.786572 5767 hybrid_clock.cc:648] HybridClock initialized: now 1790492091786509 us; error 41 us; skew 500 ppm
I20260927 06:54:51.787014 5767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:54:51.789184 5767 webserver.cc:460] Webserver started at http://127.0.97.62:41487/ using document root <none> and password file <none>
I20260927 06:54:51.789840 5767 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:51.789966 5767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:51.790323 5767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:51.793049 5767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "04208204b59745fea8fc9a410c9c2400"
format_stamp: "Formatted at 2026-09-27 06:54:51 on dist-test-slave-f892"
server_key: "7527408dc134b55942a30b0650ed613f"
server_key_iv: "83166c8a29a752c67b52f9b1800bc842"
server_key_version: "encryptionkey@0"
I20260927 06:54:51.793864 5767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "04208204b59745fea8fc9a410c9c2400"
format_stamp: "Formatted at 2026-09-27 06:54:51 on dist-test-slave-f892"
server_key: "7527408dc134b55942a30b0650ed613f"
server_key_iv: "83166c8a29a752c67b52f9b1800bc842"
server_key_version: "encryptionkey@0"
I20260927 06:54:51.798858 5767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 06:54:51.802052 5781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:51.803267 5767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 06:54:51.803447 5767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "04208204b59745fea8fc9a410c9c2400"
format_stamp: "Formatted at 2026-09-27 06:54:51 on dist-test-slave-f892"
server_key: "7527408dc134b55942a30b0650ed613f"
server_key_iv: "83166c8a29a752c67b52f9b1800bc842"
server_key_version: "encryptionkey@0"
I20260927 06:54:51.803653 5767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:51.832839 5767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:51.833983 5767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:51.834324 5767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:51.851794 5767 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36813
I20260927 06:54:51.851814 5832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36813 every 8 connection(s)
I20260927 06:54:51.853089 5767 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:54:51.856527 5833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:51.860457 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 5767
I20260927 06:54:51.860672 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:54:51.861449 388 external_mini_cluster.cc:1428] Setting key 5f0d6aa7eb1e9f736889212c7ac74b15
I20260927 06:54:51.867671 5833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400: Bootstrap starting.
I20260927 06:54:51.870975 5833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:51.872152 5833 log.cc:824] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:51.875415 5833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400: No bootstrap required, opened a new log
I20260927 06:54:51.875645 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:51.863252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.0.97.62:36813)
0927 06:54:51.864169 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:51.864179 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:51.864217 (+ 38us) server_negotiation.cc:407] Connection header received
0927 06:54:51.866321 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:51.866342 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:51.866742 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:51.867024 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:51.867588 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:51.868933 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:51.870711 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:51.871597 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:51.871745 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:51.872992 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:51.873002 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:51.873019 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:51.873732 (+ 713us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:51.873814 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:51.873834 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:51.873928 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:51.874344 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 06:54:51.874624 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":328,"threads_started":1}
I20260927 06:54:51.879246 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:54:51.883302 5833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04208204b59745fea8fc9a410c9c2400" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36813 } }
I20260927 06:54:51.883667 5833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:51.883772 5833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04208204b59745fea8fc9a410c9c2400, State: Initialized, Role: FOLLOWER
I20260927 06:54:51.884266 5833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04208204b59745fea8fc9a410c9c2400" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36813 } }
I20260927 06:54:51.884503 5833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:54:51.884640 5833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:54:51.884820 5833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:51.887197 5833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04208204b59745fea8fc9a410c9c2400" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36813 } }
I20260927 06:54:51.887687 5833 leader_election.cc:302] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04208204b59745fea8fc9a410c9c2400; no voters:
I20260927 06:54:51.888208 5833 leader_election.cc:288] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:54:51.888329 5838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:51.888999 5838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400 [term 1 LEADER]: Becoming Leader. State: Replica: 04208204b59745fea8fc9a410c9c2400, State: Running, Role: LEADER
I20260927 06:54:51.889566 5838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04208204b59745fea8fc9a410c9c2400" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36813 } }
I20260927 06:54:51.890151 5833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:54:51.893080 5839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04208204b59745fea8fc9a410c9c2400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04208204b59745fea8fc9a410c9c2400" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36813 } } }
I20260927 06:54:51.893532 5839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:51.893879 5840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04208204b59745fea8fc9a410c9c2400. Latest consensus state: current_term: 1 leader_uuid: "04208204b59745fea8fc9a410c9c2400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04208204b59745fea8fc9a410c9c2400" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36813 } } }
I20260927 06:54:51.894256 5840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:51.899001 5848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:54:51.903873 5848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:54:51.913182 5848 catalog_manager.cc:1347] Generated new cluster ID: cf76b5a0bda34bdc80176239418b7f30
I20260927 06:54:51.913290 5848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:54:51.948609 5848 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:54:51.950448 5848 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:54:51.965222 5848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04208204b59745fea8fc9a410c9c2400: Generated new TSK 0
I20260927 06:54:51.966286 5848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:54:51.988835 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:51.979563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.0.97.62:36813)
0927 06:54:51.979753 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:51.979759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:51.980386 (+ 627us) server_negotiation.cc:407] Connection header received
0927 06:54:51.980600 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:51.980609 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:51.980744 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:51.980990 (+ 246us) server_negotiation.cc:226] Negotiated authn=JWT
0927 06:54:51.981885 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:51.983214 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:51.984641 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:51.985337 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:51.985462 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:51.985970 (+ 508us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 06:54:51.988218 (+ 2248us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45285/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 06:54:51.988513 (+ 295us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38622: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45285/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 06:54:51.988965 5836 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38622: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45285/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 06:54:51.990559 5862 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:54:51.979564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36813 (local address 127.0.0.1:38622)
0927 06:54:51.980217 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:51.980233 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:51.980346 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:51.981041 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:51.981046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:51.981059 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0927 06:54:51.981574 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:51.981590 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:51.983389 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:51.983395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:51.984492 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:51.984503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:51.985453 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:51.985514 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:51.985647 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:51.985679 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 06:54:51.988419 (+ 2740us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45285/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 06:54:51.990223 (+ 1804us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:36813: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45285/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":1475,"thread_start_us":384,"threads_started":1}
I20260927 06:54:51.991815 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 5767
I20260927 06:54:51.992925 5828 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 06:54:52.004853 5863 webserver.cc:533] Webserver: sslize failed:
E20260927 06:54:52.005002 5863 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 06:54:52.086443 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 5767
2026-09-27T06:54:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (724 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T06:54:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:54:52Z Disabled control of system clock
I20260927 06:54:52.165680 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45053
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45053
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/testchainca.cert with env {}
W20260927 06:54:52.476541 5868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:52.476818 5868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:52.476900 5868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:52.483970 5868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:54:52.484066 5868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:52.484112 5868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:52.484160 5868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:54:52.484203 5868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 06:54:52.484256 5868 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 06:54:52.484333 5868 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 06:54:52.493957 5868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45053
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:52.495118 5868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:52.496245 5868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:52.507436 5873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:52.507405 5874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:52.507563 5876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:52.507964 5868 server_base.cc:1034] running on GCE node
I20260927 06:54:52.508678 5868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:52.509923 5868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:52.511122 5868 hybrid_clock.cc:648] HybridClock initialized: now 1790492092511073 us; error 49 us; skew 500 ppm
I20260927 06:54:52.511448 5868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:54:52.513635 5868 webserver.cc:460] Webserver started at http://127.0.97.62:33677/ using document root <none> and password file <none>
I20260927 06:54:52.514401 5868 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:52.514518 5868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:52.514835 5868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:52.517642 5868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "cec1ca9d16c2477e85654b822aee289a"
format_stamp: "Formatted at 2026-09-27 06:54:52 on dist-test-slave-f892"
server_key: "215ef61ec67c4ef613b0cd0e4283ed4a"
server_key_iv: "c2f02625167b7dc7ae85696f9e269ad7"
server_key_version: "encryptionkey@0"
I20260927 06:54:52.518476 5868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "cec1ca9d16c2477e85654b822aee289a"
format_stamp: "Formatted at 2026-09-27 06:54:52 on dist-test-slave-f892"
server_key: "215ef61ec67c4ef613b0cd0e4283ed4a"
server_key_iv: "c2f02625167b7dc7ae85696f9e269ad7"
server_key_version: "encryptionkey@0"
I20260927 06:54:52.523433 5868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 06:54:52.526721 5882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:52.527997 5868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 06:54:52.528236 5868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "cec1ca9d16c2477e85654b822aee289a"
format_stamp: "Formatted at 2026-09-27 06:54:52 on dist-test-slave-f892"
server_key: "215ef61ec67c4ef613b0cd0e4283ed4a"
server_key_iv: "c2f02625167b7dc7ae85696f9e269ad7"
server_key_version: "encryptionkey@0"
I20260927 06:54:52.528456 5868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:52.542680 5868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:52.543426 5868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:52.543700 5868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:52.561482 5868 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45053
I20260927 06:54:52.561491 5933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45053 every 8 connection(s)
I20260927 06:54:52.562882 5868 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:54:52.566357 5934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:52.572242 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 5868
I20260927 06:54:52.572424 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:54:52.573064 388 external_mini_cluster.cc:1428] Setting key 0b74dc34ec5664dc399ae72468a9c760
I20260927 06:54:52.576591 5934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a: Bootstrap starting.
I20260927 06:54:52.579505 5934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:52.580411 5934 log.cc:824] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:52.582983 5934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a: No bootstrap required, opened a new log
I20260927 06:54:52.585948 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:52.574822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.0.97.62:45053)
0927 06:54:52.575700 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:52.575710 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:52.575746 (+ 36us) server_negotiation.cc:407] Connection header received
0927 06:54:52.577365 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:52.577393 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:52.577823 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:52.578235 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:52.578756 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:52.580020 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:52.581272 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:52.581928 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:52.582176 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:52.583283 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:52.583292 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:52.583309 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:54:52.584014 (+ 705us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:52.584104 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:54:52.584128 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:52.584260 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:52.584427 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 06:54:52.584733 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":300,"threads_started":1}
I20260927 06:54:52.589599 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:54:52.591442 5934 raft_consensus.cc:348] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec1ca9d16c2477e85654b822aee289a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45053 } }
I20260927 06:54:52.591764 5934 raft_consensus.cc:374] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:52.591886 5934 raft_consensus.cc:729] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec1ca9d16c2477e85654b822aee289a, State: Initialized, Role: FOLLOWER
I20260927 06:54:52.592551 5934 consensus_queue.cc:260] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec1ca9d16c2477e85654b822aee289a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45053 } }
I20260927 06:54:52.592794 5934 raft_consensus.cc:388] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:54:52.592939 5934 raft_consensus.cc:482] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:54:52.593108 5934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:52.595386 5934 raft_consensus.cc:504] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec1ca9d16c2477e85654b822aee289a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45053 } }
I20260927 06:54:52.595861 5934 leader_election.cc:302] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cec1ca9d16c2477e85654b822aee289a; no voters:
I20260927 06:54:52.596413 5934 leader_election.cc:288] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:54:52.596580 5939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:52.597348 5939 raft_consensus.cc:686] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a [term 1 LEADER]: Becoming Leader. State: Replica: cec1ca9d16c2477e85654b822aee289a, State: Running, Role: LEADER
I20260927 06:54:52.597926 5939 consensus_queue.cc:237] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec1ca9d16c2477e85654b822aee289a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45053 } }
I20260927 06:54:52.598641 5934 sys_catalog.cc:564] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:54:52.600822 5940 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cec1ca9d16c2477e85654b822aee289a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec1ca9d16c2477e85654b822aee289a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45053 } } }
I20260927 06:54:52.601250 5940 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:52.606684 5941 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cec1ca9d16c2477e85654b822aee289a. Latest consensus state: current_term: 1 leader_uuid: "cec1ca9d16c2477e85654b822aee289a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec1ca9d16c2477e85654b822aee289a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45053 } } }
I20260927 06:54:52.606976 5941 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:52.607620 5948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:54:52.611682 5948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:54:52.619354 5948 catalog_manager.cc:1347] Generated new cluster ID: 4c40b116b2314475b7cedaf76bc3194b
I20260927 06:54:52.619460 5948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:54:52.643893 5948 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:54:52.645344 5948 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:54:52.655383 5948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cec1ca9d16c2477e85654b822aee289a: Generated new TSK 0
I20260927 06:54:52.656384 5948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:54:52.706748 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:52.679038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.0.97.62:45053)
0927 06:54:52.679298 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:52.679307 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:52.679720 (+ 413us) server_negotiation.cc:407] Connection header received
0927 06:54:52.679924 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:52.679928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:52.680000 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:52.680125 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0927 06:54:52.680633 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:52.681328 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:52.683384 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:52.683852 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:52.683950 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:52.684487 (+ 537us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 06:54:52.706220 (+ 21733us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 06:54:52.706522 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37816: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 06:54:52.706897 5937 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37816: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 06:54:52.707150 5963 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:54:52.679001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45053 (local address 127.0.0.1:37816)
0927 06:54:52.679613 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:52.679633 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:52.679778 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:52.680162 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:52.680167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:52.680180 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0927 06:54:52.680493 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:52.680504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:52.681475 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:52.681481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:52.683249 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:52.683263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:52.683974 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:52.684033 (+ 59us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:52.684218 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:52.684262 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 06:54:52.706466 (+ 22204us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 06:54:52.706743 (+ 277us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:45053: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":322,"threads_started":1}
I20260927 06:54:52.708657 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 5868
I20260927 06:54:52.709753 5929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:52.801941 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 5868
2026-09-27T06:54:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (695 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (393 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (392 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:54:53 dist-test-slave-f892 krb5kdc[5978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:54:53 dist-test-slave-f892 krb5kdc[5978](info): set up 2 sockets
Sep 27 06:54:53 dist-test-slave-f892 krb5kdc[5978](info): commencing operation
krb5kdc: starting...
W20260927 06:54:54.674636 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:54:54 dist-test-slave-f892 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:54:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:54:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:54:54.865643 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36107
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:55.174587 5991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:55.174845 5991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:55.174929 5991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:55.181792 5991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:54:55.181879 5991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:55.181922 5991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:55.181995 5991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:54:55.182093 5991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:54:55.191272 5991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:55.192396 5991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:55.193634 5991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:55.205075 5997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:55.205020 5999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:55.205075 5996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:55.205770 5991 server_base.cc:1034] running on GCE node
I20260927 06:54:55.206609 5991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:55.207867 5991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:55.209089 5991 hybrid_clock.cc:648] HybridClock initialized: now 1790492095209051 us; error 53 us; skew 500 ppm
I20260927 06:54:55.209412 5991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:55 dist-test-slave-f892 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:55.212309 5991 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:54:55.213579 5991 webserver.cc:460] Webserver started at http://127.0.97.62:37385/ using document root <none> and password file <none>
I20260927 06:54:55.214294 5991 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:55.214435 5991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:55.214728 5991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:55.217540 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "dd218128eec147bfb9d4a0a7feb8206b"
format_stamp: "Formatted at 2026-09-27 06:54:55 on dist-test-slave-f892"
server_key: "6e183d06e01d744f981b71098306a077"
server_key_iv: "3f26f261ff7f5157bd7b379c94f60320"
server_key_version: "encryptionkey@0"
I20260927 06:54:55.218358 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "dd218128eec147bfb9d4a0a7feb8206b"
format_stamp: "Formatted at 2026-09-27 06:54:55 on dist-test-slave-f892"
server_key: "6e183d06e01d744f981b71098306a077"
server_key_iv: "3f26f261ff7f5157bd7b379c94f60320"
server_key_version: "encryptionkey@0"
I20260927 06:54:55.223182 5991 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 06:54:55.226361 6006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:55.227605 5991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 06:54:55.227772 5991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "dd218128eec147bfb9d4a0a7feb8206b"
format_stamp: "Formatted at 2026-09-27 06:54:55 on dist-test-slave-f892"
server_key: "6e183d06e01d744f981b71098306a077"
server_key_iv: "3f26f261ff7f5157bd7b379c94f60320"
server_key_version: "encryptionkey@0"
I20260927 06:54:55.227964 5991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:55.251873 5991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:55.255539 5991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:55.255806 5991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:55.273247 5991 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36107
I20260927 06:54:55.273281 6057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36107 every 8 connection(s)
I20260927 06:54:55.274561 5991 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:54:55.278367 6058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:55.283049 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 5991
I20260927 06:54:55.283257 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:54:55.284047 388 external_mini_cluster.cc:1428] Setting key 4432172cca375e65b2315b23a92c8a5d
I20260927 06:54:55.291682 6058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b: Bootstrap starting.
I20260927 06:54:55.294796 6058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:55.295805 6058 log.cc:824] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:55.298324 6058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b: No bootstrap required, opened a new log
Sep 27 06:54:55 dist-test-slave-f892 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:55.305534 6058 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd218128eec147bfb9d4a0a7feb8206b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36107 } }
I20260927 06:54:55.305872 6058 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:55.305985 6058 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd218128eec147bfb9d4a0a7feb8206b, State: Initialized, Role: FOLLOWER
I20260927 06:54:55.306638 6058 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd218128eec147bfb9d4a0a7feb8206b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36107 } }
I20260927 06:54:55.306896 6058 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:54:55.307053 6058 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:54:55.307228 6058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:55.308636 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:55.286065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58530 (local address 127.0.97.62:36107)
0927 06:54:55.290735 (+ 4670us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:55.290747 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:55.290786 (+ 39us) server_negotiation.cc:407] Connection header received
0927 06:54:55.292492 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:55.292512 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:55.292850 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:55.293224 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:55.294696 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:55.296030 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:55.297202 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:55.297823 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:55.298043 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:55.301798 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:55.301805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:55.301822 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:55.301856 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:55.304847 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:55.305471 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:55.305477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:55.305482 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:55.305625 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:55.306095 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:55.306098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:55.306100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:55.306604 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:55.306934 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:55.307300 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 06:54:55.307676 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4442,"thread_start_us":4210,"threads_started":1}
I20260927 06:54:55.309619 6058 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd218128eec147bfb9d4a0a7feb8206b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36107 } }
I20260927 06:54:55.310159 6058 leader_election.cc:302] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd218128eec147bfb9d4a0a7feb8206b; no voters:
I20260927 06:54:55.310796 6058 leader_election.cc:288] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:54:55.310909 6063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:55.311952 6063 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b [term 1 LEADER]: Becoming Leader. State: Replica: dd218128eec147bfb9d4a0a7feb8206b, State: Running, Role: LEADER
I20260927 06:54:55.312510 6063 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd218128eec147bfb9d4a0a7feb8206b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36107 } }
I20260927 06:54:55.312927 6058 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:54:55.315200 6065 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd218128eec147bfb9d4a0a7feb8206b. Latest consensus state: current_term: 1 leader_uuid: "dd218128eec147bfb9d4a0a7feb8206b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd218128eec147bfb9d4a0a7feb8206b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36107 } } }
I20260927 06:54:55.315486 6064 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd218128eec147bfb9d4a0a7feb8206b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd218128eec147bfb9d4a0a7feb8206b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36107 } } }
I20260927 06:54:55.315733 6065 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:55.315804 6064 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:55.316686 6071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:54:55.322388 6071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:54:55.330181 6071 catalog_manager.cc:1347] Generated new cluster ID: c83dedf095f74258a7903cd402e1680c
I20260927 06:54:55.330288 6071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:54:55.356035 6071 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:54:55.357402 6071 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:54:55.376894 6071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd218128eec147bfb9d4a0a7feb8206b: Generated new TSK 0
I20260927 06:54:55.377698 6071 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 06:54:55.431988 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36107
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:55.758162 6084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:55.758427 6084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:55.758517 6084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:55.765617 6084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:55.765717 6084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:55.765821 6084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:54:55.775000 6084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:55.776096 6084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:55.777352 6084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:55.789609 6089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:55.789525 6090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:55.789565 6092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:55.790623 6084 server_base.cc:1034] running on GCE node
I20260927 06:54:55.791162 6084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:55.792119 6084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:55.793352 6084 hybrid_clock.cc:648] HybridClock initialized: now 1790492095793317 us; error 48 us; skew 500 ppm
I20260927 06:54:55.793651 6084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:55 dist-test-slave-f892 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:55.796804 6084 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:54:55.798310 6084 webserver.cc:460] Webserver started at http://127.0.97.1:39083/ using document root <none> and password file <none>
I20260927 06:54:55.798967 6084 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:55.799110 6084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:55.799418 6084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:55.802440 6084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "e9f69d50b67a4a2ebf91f6d09b45691c"
format_stamp: "Formatted at 2026-09-27 06:54:55 on dist-test-slave-f892"
server_key: "fddff58da4b7fdbf22f7c88a99920776"
server_key_iv: "2597a9f819f6dbd4455aae75acb53384"
server_key_version: "encryptionkey@0"
I20260927 06:54:55.803196 6084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e9f69d50b67a4a2ebf91f6d09b45691c"
format_stamp: "Formatted at 2026-09-27 06:54:55 on dist-test-slave-f892"
server_key: "fddff58da4b7fdbf22f7c88a99920776"
server_key_iv: "2597a9f819f6dbd4455aae75acb53384"
server_key_version: "encryptionkey@0"
I20260927 06:54:55.808030 6084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 06:54:55.811291 6099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:55.812573 6084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 06:54:55.812733 6084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "e9f69d50b67a4a2ebf91f6d09b45691c"
format_stamp: "Formatted at 2026-09-27 06:54:55 on dist-test-slave-f892"
server_key: "fddff58da4b7fdbf22f7c88a99920776"
server_key_iv: "2597a9f819f6dbd4455aae75acb53384"
server_key_version: "encryptionkey@0"
I20260927 06:54:55.812922 6084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:55.834227 6084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:55.838306 6084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:55.838586 6084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:55.839519 6084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:55.840806 6084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:55.840909 6084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:55.841053 6084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:55.841122 6084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:55.872188 6084 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40947
I20260927 06:54:55.872293 6211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40947 every 8 connection(s)
I20260927 06:54:55.873461 6084 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:54:55.883301 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 6084
I20260927 06:54:55.883538 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:54:55.884153 388 external_mini_cluster.cc:1428] Setting key d7f5dfa78e9dd79508dde2a0b3b82d5c
Sep 27 06:54:55 dist-test-slave-f892 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:55.893528 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:55.876094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50397 (local address 127.0.97.62:36107)
0927 06:54:55.876487 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:55.876495 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:55.877474 (+ 979us) server_negotiation.cc:407] Connection header received
0927 06:54:55.879444 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:55.879452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:55.879533 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:55.879685 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:55.881156 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:55.881857 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:55.883327 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:55.883858 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:55.883992 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:55.887350 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:55.887355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:55.887358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:55.887386 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:55.890387 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:55.891350 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:55.891354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:55.891356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:55.891473 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:55.891914 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:55.891924 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:55.891926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:55.892169 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:55.892334 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:55.892971 (+ 637us) server_negotiation.cc:299] Negotiation successful
0927 06:54:55.893173 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":178,"threads_started":1}
I20260927 06:54:55.894419 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:55.876602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36107 (local address 127.0.97.1:50397)
0927 06:54:55.877328 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:55.877369 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:55.879095 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:55.879969 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:55.879976 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:55.880471 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:55.880975 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:55.880996 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:55.882024 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:55.882030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:55.883173 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:55.883184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:55.884072 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:55.884081 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:55.884361 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:55.884990 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:55.885020 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:55.887134 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:55.890523 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:55.890534 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:55.890552 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:55.891200 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:55.891610 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:55.891614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:55.891616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:55.891795 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:55.892316 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:55.892323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:55.892614 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 06:54:55.892868 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 06:54:55.893333 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":287,"threads_started":1}
I20260927 06:54:55.895859 6212 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36107
I20260927 06:54:55.896132 6212 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:55.896763 6212 heartbeater.cc:511] Master 127.0.97.62:36107 requested a full tablet report, sending...
I20260927 06:54:55.899003 6023 ts_manager.cc:194] Registered new tserver with Master: e9f69d50b67a4a2ebf91f6d09b45691c (127.0.97.1:40947)
I20260927 06:54:55.901070 6023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50397
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 06:54:55.927356 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36107
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:56.236704 6218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:56.236953 6218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:56.237051 6218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:56.244088 6218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:56.244189 6218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:56.244307 6218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:54:56.253708 6218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:56.254866 6218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:56.255991 6218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:56.270524 6226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:56.270494 6223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:56.270509 6224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:56.271456 6218 server_base.cc:1034] running on GCE node
I20260927 06:54:56.272037 6218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:56.272957 6218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:56.274164 6218 hybrid_clock.cc:648] HybridClock initialized: now 1790492096274080 us; error 83 us; skew 500 ppm
I20260927 06:54:56.274442 6218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:56 dist-test-slave-f892 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:56.277438 6218 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:54:56.283430 6218 webserver.cc:460] Webserver started at http://127.0.97.2:42459/ using document root <none> and password file <none>
I20260927 06:54:56.284125 6218 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:56.284272 6218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:56.284561 6218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:56.287431 6218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "567ce615091048d6a71d266c5c20a680"
format_stamp: "Formatted at 2026-09-27 06:54:56 on dist-test-slave-f892"
server_key: "b1bcce90ec1f08b22b8b116a91239ad4"
server_key_iv: "bbb7a766f434e4d5f96a40c7eb57e59c"
server_key_version: "encryptionkey@0"
I20260927 06:54:56.288180 6218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "567ce615091048d6a71d266c5c20a680"
format_stamp: "Formatted at 2026-09-27 06:54:56 on dist-test-slave-f892"
server_key: "b1bcce90ec1f08b22b8b116a91239ad4"
server_key_iv: "bbb7a766f434e4d5f96a40c7eb57e59c"
server_key_version: "encryptionkey@0"
I20260927 06:54:56.294179 6218 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 06:54:56.297468 6233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:56.298719 6218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:54:56.298885 6218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "567ce615091048d6a71d266c5c20a680"
format_stamp: "Formatted at 2026-09-27 06:54:56 on dist-test-slave-f892"
server_key: "b1bcce90ec1f08b22b8b116a91239ad4"
server_key_iv: "bbb7a766f434e4d5f96a40c7eb57e59c"
server_key_version: "encryptionkey@0"
I20260927 06:54:56.299090 6218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:56.325341 6218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:56.329594 6218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:56.329879 6218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:56.330832 6218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:56.332145 6218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:56.332249 6218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:56.332412 6218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:56.332484 6218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:56.364236 6218 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43027
I20260927 06:54:56.364284 6345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43027 every 8 connection(s)
I20260927 06:54:56.365463 6218 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:54:56.375046 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 6218
I20260927 06:54:56.375212 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:54:56.375597 388 external_mini_cluster.cc:1428] Setting key 9b96e4bac635229801a13b40bb09b0fe
Sep 27 06:54:56 dist-test-slave-f892 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:56.384567 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:56.367930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47071 (local address 127.0.97.62:36107)
0927 06:54:56.368137 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:56.368143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:56.369158 (+ 1015us) server_negotiation.cc:407] Connection header received
0927 06:54:56.370830 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:56.370834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:56.370898 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:56.370986 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:56.372285 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:56.372989 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:56.374489 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:56.374904 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:56.375032 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:56.378912 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:56.378916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:56.378920 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:56.378949 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:56.381346 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:56.382281 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:56.382285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:56.382287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:56.382404 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:56.382850 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:56.382853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:56.382855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:56.383088 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:56.383191 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:56.383891 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 06:54:56.384134 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 06:54:56.385018 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:56.368344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36107 (local address 127.0.97.2:47071)
0927 06:54:56.369004 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:56.369044 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:56.370606 (+ 1562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:56.371164 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:56.371171 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:56.371625 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:56.372139 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:56.372154 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:56.373107 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:56.373111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:56.374359 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:56.374373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:56.375113 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:56.375120 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:56.375392 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:56.376053 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:56.376073 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:56.378707 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:56.381478 (+ 2771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:56.381489 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:56.381505 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:56.382168 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:56.382518 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:56.382526 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:56.382533 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:56.382745 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:56.383209 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:56.383220 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:56.383545 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 06:54:56.383744 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 06:54:56.384059 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":271,"threads_started":1}
I20260927 06:54:56.386579 6346 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36107
I20260927 06:54:56.386842 6346 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:56.387478 6346 heartbeater.cc:511] Master 127.0.97.62:36107 requested a full tablet report, sending...
I20260927 06:54:56.388842 6023 ts_manager.cc:194] Registered new tserver with Master: 567ce615091048d6a71d266c5c20a680 (127.0.97.2:43027)
I20260927 06:54:56.390081 6023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47071
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 06:54:56.421856 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36107
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:56.731495 6351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:56.731786 6351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:56.731948 6351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:56.739054 6351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:56.739197 6351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:56.739367 6351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:54:56.748489 6351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:56.749696 6351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:56.751140 6351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:56.763731 6357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:56.763693 6359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:56.763830 6356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:56.764705 6351 server_base.cc:1034] running on GCE node
I20260927 06:54:56.765304 6351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:56.766239 6351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:56.767488 6351 hybrid_clock.cc:648] HybridClock initialized: now 1790492096767446 us; error 36 us; skew 500 ppm
I20260927 06:54:56.767831 6351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:56 dist-test-slave-f892 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:56.770753 6351 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:54:56.772063 6351 webserver.cc:460] Webserver started at http://127.0.97.3:42587/ using document root <none> and password file <none>
I20260927 06:54:56.772706 6351 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:56.772857 6351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:56.773147 6351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:56.775970 6351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "7ceb601c19894463961105b825dda925"
format_stamp: "Formatted at 2026-09-27 06:54:56 on dist-test-slave-f892"
server_key: "2e1c0f80d29f90151b7d110292324bdb"
server_key_iv: "f0b934efd801cdc7b4d139ba05223f6a"
server_key_version: "encryptionkey@0"
I20260927 06:54:56.776726 6351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7ceb601c19894463961105b825dda925"
format_stamp: "Formatted at 2026-09-27 06:54:56 on dist-test-slave-f892"
server_key: "2e1c0f80d29f90151b7d110292324bdb"
server_key_iv: "f0b934efd801cdc7b4d139ba05223f6a"
server_key_version: "encryptionkey@0"
I20260927 06:54:56.781507 6351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 06:54:56.784606 6366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:56.785797 6351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 06:54:56.785959 6351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "7ceb601c19894463961105b825dda925"
format_stamp: "Formatted at 2026-09-27 06:54:56 on dist-test-slave-f892"
server_key: "2e1c0f80d29f90151b7d110292324bdb"
server_key_iv: "f0b934efd801cdc7b4d139ba05223f6a"
server_key_version: "encryptionkey@0"
I20260927 06:54:56.786193 6351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:56.804353 6351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:56.808039 6351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:56.808313 6351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:56.809197 6351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:56.810456 6351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:56.810554 6351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:56.810694 6351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:56.810763 6351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:56.842202 6351 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38011
I20260927 06:54:56.842237 6478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38011 every 8 connection(s)
I20260927 06:54:56.843366 6351 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:54:56.849289 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 6351
I20260927 06:54:56.849475 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:54:56.849864 388 external_mini_cluster.cc:1428] Setting key 043625aaf8b5ba3f31573b28b81861f1
Sep 27 06:54:56 dist-test-slave-f892 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:54:56.861224 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:56.845761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50341 (local address 127.0.97.62:36107)
0927 06:54:56.845994 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:56.846000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:56.847079 (+ 1079us) server_negotiation.cc:407] Connection header received
0927 06:54:56.848665 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:56.848670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:56.848732 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:56.848828 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:56.850355 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:56.851246 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:56.852959 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:56.853334 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:56.853431 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:56.856545 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:56.856549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:56.856552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:56.856580 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:56.858854 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:56.859473 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:56.859476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:56.859479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:56.859590 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:56.859934 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:56.859937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:56.859939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:56.860172 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:56.860271 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:56.860841 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 06:54:56.860993 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 06:54:56.862144 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:56.846292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36107 (local address 127.0.97.3:50341)
0927 06:54:56.846933 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:56.846967 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:56.848444 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:56.849011 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:56.849018 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:56.849533 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:56.850176 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:56.850193 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:56.851414 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:56.851418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:56.852834 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:56.852845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:56.853506 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:56.853510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:56.853683 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:56.854396 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:56.854421 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:56.856363 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:56.858974 (+ 2611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:56.858980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:56.858990 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:56.859373 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:56.859674 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:56.859677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:56.859679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:56.859835 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:56.860265 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:56.860270 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:56.860538 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 06:54:56.860743 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 06:54:56.861071 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":257,"threads_started":1}
I20260927 06:54:56.863509 6479 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36107
I20260927 06:54:56.863790 6479 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:56.864475 6479 heartbeater.cc:511] Master 127.0.97.62:36107 requested a full tablet report, sending...
I20260927 06:54:56.865674 6023 ts_manager.cc:194] Registered new tserver with Master: 7ceb601c19894463961105b825dda925 (127.0.97.3:38011)
I20260927 06:54:56.866456 6023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50341
I20260927 06:54:56.877341 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/test-admin.keytab.
Sep 27 06:54:56 dist-test-slave-f892 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:56.903789 6212 heartbeater.cc:503] Master 127.0.97.62:36107 was elected leader, sending a full tablet report...
I20260927 06:54:56.904807 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 06:54:56.909587 6486 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:54:56.907467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38696 (local address 127.0.97.1:40947)
0927 06:54:56.908071 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:56.908078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:56.908101 (+ 23us) server_negotiation.cc:407] Connection header received
0927 06:54:56.908156 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:56.908163 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:56.908356 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:56.908499 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:56.909212 (+ 713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38696: BlockingRecv error: recv got EOF from 127.0.0.1:38696 (error 108)
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":334,"threads_started":1}
W20260927 06:54:56.909943 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 06:54:56 dist-test-slave-f892 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:56.911393 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 06:54:56 dist-test-slave-f892 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:56.912699 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 06:54:56 dist-test-slave-f892 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:54:56.927484 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:56.914164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.0.97.1:40947)
0927 06:54:56.914286 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:56.914290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:56.914652 (+ 362us) server_negotiation.cc:407] Connection header received
0927 06:54:56.914777 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:56.914780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:56.914830 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:56.914906 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:56.915841 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:56.916963 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:56.918289 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:56.918844 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:56.918940 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:56.922282 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:56.922292 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:56.922306 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:56.922338 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:56.924895 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:56.925518 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:56.925524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:56.925527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:56.925648 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:56.926035 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:56.926040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:56.926043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:56.926360 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:56.926646 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:56.926967 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 06:54:56.927198 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260927 06:54:56.932763 6486 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:54:56.931244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.0.97.1:40947)
0927 06:54:56.931431 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:56.931437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:56.931601 (+ 164us) server_negotiation.cc:407] Connection header received
0927 06:54:56.931764 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:56.931768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:56.931819 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:56.931891 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:56.932530 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38728: BlockingRecv error: recv got EOF from 127.0.0.1:38728 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 06:54:56.932878 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 06:54:56 dist-test-slave-f892 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:56.934309 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 06:54:56 dist-test-slave-f892 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:56.935478 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 06:54:56 dist-test-slave-f892 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:54:56.947656 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:56.936549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38734 (local address 127.0.97.1:40947)
0927 06:54:56.936730 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:56.936737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:56.936978 (+ 241us) server_negotiation.cc:407] Connection header received
0927 06:54:56.937131 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:56.937135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:56.937179 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:56.937270 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:56.938118 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:56.938769 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:56.939933 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:56.940339 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:56.940482 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:56.943518 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:56.943522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:56.943525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:56.943560 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:56.945740 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:56.946353 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:56.946356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:56.946358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:56.946471 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:56.946770 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:56.946773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:56.946775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:56.946988 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:56.947159 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:56.947333 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 06:54:56.947471 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 06:54:56.951277 6486 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:54:56.949857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.0.97.1:40947)
0927 06:54:56.950067 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:56.950072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:56.950264 (+ 192us) server_negotiation.cc:407] Connection header received
0927 06:54:56.950363 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:56.950366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:56.950444 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:56.950517 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:56.951063 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38748: BlockingRecv error: recv got EOF from 127.0.0.1:38748 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 06:54:56.951622 388 init.cc:244] Error while opening credential cache '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492047466241-388-0/krb5kdc/krb5cc)
Sep 27 06:54:56 dist-test-slave-f892 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:56.952894 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 06:54:56 dist-test-slave-f892 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:56.954097 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 06:54:56 dist-test-slave-f892 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:54:56.966234 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:56.955317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38752 (local address 127.0.97.1:40947)
0927 06:54:56.955491 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:56.955495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:56.955593 (+ 98us) server_negotiation.cc:407] Connection header received
0927 06:54:56.955691 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:56.955694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:56.955739 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:56.955809 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:56.956656 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:56.957328 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:56.958562 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:56.958952 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:56.959040 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:56.961985 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:56.961989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:56.961992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:56.962070 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:56.964214 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:56.964747 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:56.964750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:56.964752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:56.964877 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:56.965168 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:56.965171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:56.965173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:56.965370 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:56.965605 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:56.965738 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 06:54:56.965900 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 06:54:56.967635 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 6084
I20260927 06:54:56.978494 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:56.968279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38768 (local address 127.0.97.1:40947)
0927 06:54:56.968408 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:56.968413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:56.968770 (+ 357us) server_negotiation.cc:407] Connection header received
0927 06:54:56.968890 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:56.968893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:56.968946 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:56.969016 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:56.969823 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:56.970530 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:56.971633 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:56.972017 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:56.972101 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:56.973841 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:56.973847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:56.973854 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:56.973893 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:56.976377 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:56.976925 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:56.976928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:56.976930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:56.977041 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:56.977396 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:56.977399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:56.977401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:56.977614 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:56.977808 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:56.978114 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 06:54:56.978264 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 06:54:56.979012 6207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:57.061201 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 6084
I20260927 06:54:57.077565 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 6218
Sep 27 06:54:57 dist-test-slave-f892 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:54:57.091995 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:57.078404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47396 (local address 127.0.97.2:43027)
0927 06:54:57.078988 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:57.078995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:57.079019 (+ 24us) server_negotiation.cc:407] Connection header received
0927 06:54:57.079077 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:57.079084 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:57.079281 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:57.079403 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:57.080225 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:57.081209 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:57.082452 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:57.083007 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:57.083107 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:57.086783 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:57.086793 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:57.086807 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:57.086837 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:57.089305 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:57.090039 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:57.090051 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:57.090057 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:57.090201 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:57.090585 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:57.090592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:57.090596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:57.090948 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:57.091164 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:57.091531 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 06:54:57.091715 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":284,"threads_started":1}
I20260927 06:54:57.093185 6341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:57.173153 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 6218
I20260927 06:54:57.188162 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 6351
Sep 27 06:54:57 dist-test-slave-f892 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 06:54:57.202955 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:57.189041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35148 (local address 127.0.97.3:38011)
0927 06:54:57.189665 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:57.189671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:57.189699 (+ 28us) server_negotiation.cc:407] Connection header received
0927 06:54:57.189756 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:57.189762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:57.189977 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:57.190173 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:57.191095 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:57.192052 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:57.193206 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:57.193722 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:57.193813 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:57.197486 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:57.197497 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:57.197510 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:57.197563 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:57.200363 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:57.201023 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:57.201031 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:57.201033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:57.201159 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:57.201515 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:57.201521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:57.201524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:57.201836 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:57.202177 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:57.202418 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 06:54:57.202662 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":350,"threads_started":1}
I20260927 06:54:57.204128 6474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:57.290174 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 6351
I20260927 06:54:57.306535 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 5991
I20260927 06:54:57.307614 6053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:54:57.405329 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 5991
2026-09-27T06:54:57Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3822 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:54:57 dist-test-slave-f892 krb5kdc[6506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:54:57 dist-test-slave-f892 krb5kdc[6506](info): set up 2 sockets
Sep 27 06:54:57 dist-test-slave-f892 krb5kdc[6506](info): commencing operation
krb5kdc: starting...
W20260927 06:54:58.486836 388 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:54:58 dist-test-slave-f892 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:54:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260927 06:54:58.679006 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44789
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:58.995294 6519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:58.995615 6519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:58.995777 6519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:59.002967 6519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:54:59.003099 6519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:59.003196 6519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:59.003299 6519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:54:59.003369 6519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:54:59.012902 6519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:59.014194 6519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:59.015427 6519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:59.026605 6525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:59.026670 6527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:59.026647 6524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:59.027637 6519 server_base.cc:1034] running on GCE node
I20260927 06:54:59.028364 6519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:59.029582 6519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:59.030768 6519 hybrid_clock.cc:648] HybridClock initialized: now 1790492099030730 us; error 28 us; skew 500 ppm
I20260927 06:54:59.031096 6519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:59 dist-test-slave-f892 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492099, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:59.037348 6519 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260927 06:54:59.038889 6519 webserver.cc:460] Webserver started at http://127.0.97.62:36773/ using document root <none> and password file <none>
I20260927 06:54:59.039561 6519 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:59.039688 6519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:59.039999 6519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:59.042908 6519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "f346c97981314f6c97c333fb51cf4e55"
format_stamp: "Formatted at 2026-09-27 06:54:59 on dist-test-slave-f892"
server_key: "80f977662950ad32a7fdab8aac635ab9"
server_key_iv: "95534cdf9f794287fdb904ac41660117"
server_key_version: "encryptionkey@0"
I20260927 06:54:59.043661 6519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "f346c97981314f6c97c333fb51cf4e55"
format_stamp: "Formatted at 2026-09-27 06:54:59 on dist-test-slave-f892"
server_key: "80f977662950ad32a7fdab8aac635ab9"
server_key_iv: "95534cdf9f794287fdb904ac41660117"
server_key_version: "encryptionkey@0"
I20260927 06:54:59.048473 6519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 06:54:59.051605 6534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:59.052825 6519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 06:54:59.052996 6519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "f346c97981314f6c97c333fb51cf4e55"
format_stamp: "Formatted at 2026-09-27 06:54:59 on dist-test-slave-f892"
server_key: "80f977662950ad32a7fdab8aac635ab9"
server_key_iv: "95534cdf9f794287fdb904ac41660117"
server_key_version: "encryptionkey@0"
I20260927 06:54:59.053166 6519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:59.072214 6519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:59.075820 6519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:59.076113 6519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:59.093830 6519 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44789
I20260927 06:54:59.093863 6585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44789 every 8 connection(s)
I20260927 06:54:59.095211 6519 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:54:59.098731 6586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:54:59.099277 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 6519
I20260927 06:54:59.099464 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:54:59.100302 388 external_mini_cluster.cc:1428] Setting key aad35d4c037a87188dd781a086497093
I20260927 06:54:59.112555 6586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55: Bootstrap starting.
Sep 27 06:54:59 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 06:54:59.116117 6586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55: Neither blocks nor log segments found. Creating new log.
I20260927 06:54:59.117216 6586 log.cc:824] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55: Log is configured to *not* fsync() on all Append() calls
I20260927 06:54:59.120710 6586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55: No bootstrap required, opened a new log
I20260927 06:54:59.123694 6589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:59.102322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54180 (local address 127.0.97.62:44789)
0927 06:54:59.103084 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:59.103113 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:59.103170 (+ 57us) server_negotiation.cc:407] Connection header received
0927 06:54:59.105052 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:59.105093 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:59.105595 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:59.105918 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:59.107610 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:59.108839 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:59.110557 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:59.111191 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:59.111318 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:59.115694 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:59.115702 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:59.115716 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:59.115753 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:59.119323 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:59.120007 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:59.120018 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:59.120032 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:59.120233 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:59.120564 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:59.120568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:59.120571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:59.121273 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:59.121583 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:59.121885 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 06:54:59.122333 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":318,"threads_started":1}
I20260927 06:54:59.129740 6586 raft_consensus.cc:348] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } }
I20260927 06:54:59.130113 6586 raft_consensus.cc:374] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:54:59.130227 6586 raft_consensus.cc:729] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f346c97981314f6c97c333fb51cf4e55, State: Initialized, Role: FOLLOWER
I20260927 06:54:59.130741 6586 consensus_queue.cc:260] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } }
I20260927 06:54:59.130959 6586 raft_consensus.cc:388] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:54:59.131078 6586 raft_consensus.cc:482] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:54:59.131237 6586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:54:59.133247 6586 raft_consensus.cc:504] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } }
I20260927 06:54:59.133720 6586 leader_election.cc:302] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f346c97981314f6c97c333fb51cf4e55; no voters:
I20260927 06:54:59.134387 6586 leader_election.cc:288] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:54:59.134501 6591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:54:59.135298 6591 raft_consensus.cc:686] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [term 1 LEADER]: Becoming Leader. State: Replica: f346c97981314f6c97c333fb51cf4e55, State: Running, Role: LEADER
I20260927 06:54:59.135836 6591 consensus_queue.cc:237] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } }
I20260927 06:54:59.136439 6586 sys_catalog.cc:564] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:54:59.138720 6592 sys_catalog.cc:455] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } }
I20260927 06:54:59.139143 6592 sys_catalog.cc:458] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:59.138967 6593 sys_catalog.cc:455] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f346c97981314f6c97c333fb51cf4e55. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } }
I20260927 06:54:59.139266 6593 sys_catalog.cc:458] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: This master's current role is: LEADER
I20260927 06:54:59.140512 6600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:54:59.144575 6600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:54:59.152590 6600 catalog_manager.cc:1347] Generated new cluster ID: 17f0f544b36d4aad874de996e288df85
I20260927 06:54:59.152691 6600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:54:59.171566 6600 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:54:59.173359 6600 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:54:59.185335 6600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55: Generated new TSK 0
I20260927 06:54:59.186268 6600 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260927 06:54:59.246899 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44789
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:54:59.567993 6612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:54:59.568303 6612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:54:59.568463 6612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:54:59.575582 6612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:54:59.575726 6612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:54:59.575899 6612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:54:59.585182 6612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:54:59.586462 6612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:54:59.587697 6612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:54:59.600383 6617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:59.600378 6618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:54:59.600605 6620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:54:59.603273 6612 server_base.cc:1034] running on GCE node
I20260927 06:54:59.603875 6612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:54:59.604851 6612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:54:59.606101 6612 hybrid_clock.cc:648] HybridClock initialized: now 1790492099606051 us; error 39 us; skew 500 ppm
I20260927 06:54:59.606421 6612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:54:59 dist-test-slave-f892 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492099, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:54:59.609324 6612 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 06:54:59.610823 6612 webserver.cc:460] Webserver started at http://127.0.97.1:39717/ using document root <none> and password file <none>
I20260927 06:54:59.611490 6612 fs_manager.cc:359] Metadata directory not provided
I20260927 06:54:59.611630 6612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:54:59.611919 6612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:54:59.614758 6612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "8b3b556cc4054b7ca08c3abe1a70ae40"
format_stamp: "Formatted at 2026-09-27 06:54:59 on dist-test-slave-f892"
server_key: "47b6de3af3a29eacf9174a379bdc540b"
server_key_iv: "c8c6356d2a6b26e6d69162993bce3b89"
server_key_version: "encryptionkey@0"
I20260927 06:54:59.615511 6612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8b3b556cc4054b7ca08c3abe1a70ae40"
format_stamp: "Formatted at 2026-09-27 06:54:59 on dist-test-slave-f892"
server_key: "47b6de3af3a29eacf9174a379bdc540b"
server_key_iv: "c8c6356d2a6b26e6d69162993bce3b89"
server_key_version: "encryptionkey@0"
I20260927 06:54:59.620311 6612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 06:54:59.623628 6627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:59.625042 6612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 06:54:59.625193 6612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "8b3b556cc4054b7ca08c3abe1a70ae40"
format_stamp: "Formatted at 2026-09-27 06:54:59 on dist-test-slave-f892"
server_key: "47b6de3af3a29eacf9174a379bdc540b"
server_key_iv: "c8c6356d2a6b26e6d69162993bce3b89"
server_key_version: "encryptionkey@0"
I20260927 06:54:59.625448 6612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:54:59.643424 6612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:54:59.647378 6612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:54:59.647663 6612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:54:59.648546 6612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:54:59.649828 6612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:54:59.649927 6612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:59.650110 6612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:54:59.650202 6612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:54:59.682477 6612 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36787
I20260927 06:54:59.682526 6739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36787 every 8 connection(s)
I20260927 06:54:59.683727 6612 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:54:59.685750 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 6612
I20260927 06:54:59.685963 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:54:59.686797 388 external_mini_cluster.cc:1428] Setting key 6d9cf410d988b486d33d601db1f67e21
Sep 27 06:54:59 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492099, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 06:54:59.711119 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:59.686418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60749 (local address 127.0.97.62:44789)
0927 06:54:59.686890 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 06:54:59.686898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:54:59.687952 (+ 1054us) server_negotiation.cc:407] Connection header received
0927 06:54:59.690467 (+ 2515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:54:59.690471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:54:59.690527 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:54:59.690615 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:54:59.692657 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:59.693831 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:59.696404 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:54:59.697073 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:54:59.697227 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:59.701938 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:54:59.701948 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:54:59.701956 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:54:59.702022 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:54:59.706316 (+ 4294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:59.708133 (+ 1817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:59.708140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:59.708144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:59.708309 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:54:59.708949 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:54:59.708954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:54:59.708957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:54:59.709345 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:54:59.709501 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 06:54:59.710460 (+ 959us) server_negotiation.cc:299] Negotiation successful
0927 06:54:59.710693 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":223,"threads_started":1}
I20260927 06:54:59.712316 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:54:59.687012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.1:60749)
0927 06:54:59.687787 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 06:54:59.687835 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 06:54:59.690184 (+ 2349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:54:59.690901 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:54:59.690925 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:54:59.691669 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:54:59.692447 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:59.692472 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:59.694074 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:59.694082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:59.696242 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:54:59.696259 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:54:59.697325 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:54:59.697334 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:54:59.697659 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:54:59.698520 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:54:59.698559 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:54:59.701702 (+ 3143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:54:59.706465 (+ 4763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:59.706477 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:59.706496 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:59.707802 (+ 1306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:59.708546 (+ 744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:54:59.708553 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:54:59.708557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:54:59.708802 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:54:59.709494 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:54:59.709502 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:54:59.709925 (+ 423us) client_negotiation.cc:769] Sending connection context
0927 06:54:59.710283 (+ 358us) client_negotiation.cc:240] Negotiation successful
0927 06:54:59.710835 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":312,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260927 06:54:59.715808 6740 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44789
I20260927 06:54:59.716187 6740 heartbeater.cc:464] Registering TS with master...
I20260927 06:54:59.717125 6740 heartbeater.cc:511] Master 127.0.97.62:44789 requested a full tablet report, sending...
I20260927 06:54:59.719753 6551 ts_manager.cc:194] Registered new tserver with Master: 8b3b556cc4054b7ca08c3abe1a70ae40 (127.0.97.1:36787)
I20260927 06:54:59.721726 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260927 06:54:59.745523 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44789
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:55:00.055465 6746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:00.055783 6746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:00.055949 6746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:00.063113 6746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:00.063264 6746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:00.063472 6746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:55:00.073184 6746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:00.074527 6746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:00.075826 6746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:00.088732 6752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:00.088876 6751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:00.088876 6754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:00.089798 6746 server_base.cc:1034] running on GCE node
I20260927 06:55:00.090462 6746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:00.091460 6746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:00.092795 6746 hybrid_clock.cc:648] HybridClock initialized: now 1790492100092751 us; error 38 us; skew 500 ppm
I20260927 06:55:00.093103 6746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:00 dist-test-slave-f892 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492100, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:00.096200 6746 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 06:55:00.097579 6746 webserver.cc:460] Webserver started at http://127.0.97.2:36849/ using document root <none> and password file <none>
I20260927 06:55:00.098295 6746 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:00.098421 6746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:00.098709 6746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:00.101358 6746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "151233c285ec4937b55164d138143479"
format_stamp: "Formatted at 2026-09-27 06:55:00 on dist-test-slave-f892"
server_key: "56223d1bbbf2aac48ad4ec4dd8557cc5"
server_key_iv: "0531d277826a4dd0d0e7c88d9162b7e3"
server_key_version: "encryptionkey@0"
I20260927 06:55:00.102191 6746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "151233c285ec4937b55164d138143479"
format_stamp: "Formatted at 2026-09-27 06:55:00 on dist-test-slave-f892"
server_key: "56223d1bbbf2aac48ad4ec4dd8557cc5"
server_key_iv: "0531d277826a4dd0d0e7c88d9162b7e3"
server_key_version: "encryptionkey@0"
I20260927 06:55:00.107100 6746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 06:55:00.110352 6761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:00.111522 6746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 06:55:00.111686 6746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "151233c285ec4937b55164d138143479"
format_stamp: "Formatted at 2026-09-27 06:55:00 on dist-test-slave-f892"
server_key: "56223d1bbbf2aac48ad4ec4dd8557cc5"
server_key_iv: "0531d277826a4dd0d0e7c88d9162b7e3"
server_key_version: "encryptionkey@0"
I20260927 06:55:00.111874 6746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:00.124459 6746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:00.129174 6746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:00.129488 6746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:00.130553 6746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:00.131974 6746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:00.132053 6746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:00.132206 6746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:00.132295 6746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:00.163054 6746 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33493
I20260927 06:55:00.163110 6873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33493 every 8 connection(s)
I20260927 06:55:00.164289 6746 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:55:00.173202 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 6746
I20260927 06:55:00.173420 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:55:00.173890 388 external_mini_cluster.cc:1428] Setting key 7c08173191d880eea0fec667f27f56ef
Sep 27 06:55:00 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492100, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 06:55:00.184437 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:00.166895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60305 (local address 127.0.97.62:44789)
0927 06:55:00.167066 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:00.167071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:00.168158 (+ 1087us) server_negotiation.cc:407] Connection header received
0927 06:55:00.169862 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:00.169866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:00.169948 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:00.170118 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:00.171516 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:00.172212 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:00.173798 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:00.174413 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:00.174536 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:00.178132 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:00.178141 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:00.178150 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:00.178193 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:00.181598 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:00.182405 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:00.182409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:00.182412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:00.182525 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:00.182974 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:00.182977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:00.182979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:00.183194 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:00.183283 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:00.184056 (+ 773us) server_negotiation.cc:299] Negotiation successful
0927 06:55:00.184203 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 06:55:00.185905 6876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:00.167312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.2:60305)
0927 06:55:00.167996 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:00.168035 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:00.169598 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:00.170312 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:00.170320 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:00.170805 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:00.171342 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:00.171358 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:00.172355 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:00.172358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:00.173658 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:00.173669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:00.174641 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:00.174651 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:00.174893 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:00.175784 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:00.175814 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:00.177878 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:00.181735 (+ 3857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:00.181741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:00.181753 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:00.182297 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:00.182637 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:00.182643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:00.182648 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:00.182874 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:00.183313 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:00.183322 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:00.183733 (+ 411us) client_negotiation.cc:769] Sending connection context
0927 06:55:00.184004 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 06:55:00.184519 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":269,"threads_started":1}
I20260927 06:55:00.187288 6874 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44789
I20260927 06:55:00.187601 6874 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:00.188274 6874 heartbeater.cc:511] Master 127.0.97.62:44789 requested a full tablet report, sending...
I20260927 06:55:00.189683 6551 ts_manager.cc:194] Registered new tserver with Master: 151233c285ec4937b55164d138143479 (127.0.97.2:33493)
I20260927 06:55:00.190902 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60305
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 06:55:00.221303 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44789
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:55:00.533439 6879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:00.533753 6879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:00.533870 6879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:00.541148 6879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:00.541239 6879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:00.541425 6879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:55:00.550494 6879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:00.551676 6879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:00.552889 6879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:00.565675 6884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:00.565860 6885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:00.565872 6887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:00.566856 6879 server_base.cc:1034] running on GCE node
I20260927 06:55:00.567417 6879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:00.568287 6879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:00.569516 6879 hybrid_clock.cc:648] HybridClock initialized: now 1790492100569473 us; error 62 us; skew 500 ppm
I20260927 06:55:00.569800 6879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:00 dist-test-slave-f892 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492100, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:00.572674 6879 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 06:55:00.574103 6879 webserver.cc:460] Webserver started at http://127.0.97.3:33639/ using document root <none> and password file <none>
I20260927 06:55:00.574725 6879 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:00.574874 6879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:00.575157 6879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:00.577890 6879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "9eebd909c11f4326bb832093502f823f"
format_stamp: "Formatted at 2026-09-27 06:55:00 on dist-test-slave-f892"
server_key: "b4a7b9aa97caf888bf51c5f6286dbfac"
server_key_iv: "92c68ce4239fc0f78bb37163715ab994"
server_key_version: "encryptionkey@0"
I20260927 06:55:00.578732 6879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9eebd909c11f4326bb832093502f823f"
format_stamp: "Formatted at 2026-09-27 06:55:00 on dist-test-slave-f892"
server_key: "b4a7b9aa97caf888bf51c5f6286dbfac"
server_key_iv: "92c68ce4239fc0f78bb37163715ab994"
server_key_version: "encryptionkey@0"
I20260927 06:55:00.583644 6879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 06:55:00.586782 6894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:00.587929 6879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 06:55:00.588116 6879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "9eebd909c11f4326bb832093502f823f"
format_stamp: "Formatted at 2026-09-27 06:55:00 on dist-test-slave-f892"
server_key: "b4a7b9aa97caf888bf51c5f6286dbfac"
server_key_iv: "92c68ce4239fc0f78bb37163715ab994"
server_key_version: "encryptionkey@0"
I20260927 06:55:00.588300 6879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:00.604857 6879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:00.608814 6879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:00.609095 6879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:00.610072 6879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:00.611358 6879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:00.611431 6879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:00.611516 6879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:00.611560 6879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:00.643985 6879 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41593
I20260927 06:55:00.644042 7006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41593 every 8 connection(s)
I20260927 06:55:00.645229 6879 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:55:00.649958 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 6879
I20260927 06:55:00.650177 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:55:00.650604 388 external_mini_cluster.cc:1428] Setting key 9e8d9380bde0d2a2957befdc02479586
Sep 27 06:55:00 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492100, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 06:55:00.664095 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:00.647633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46099 (local address 127.0.97.62:44789)
0927 06:55:00.647823 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:00.647829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:00.648950 (+ 1121us) server_negotiation.cc:407] Connection header received
0927 06:55:00.650788 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:00.650794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:00.650877 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:00.650990 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:00.652842 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:00.653676 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:00.655194 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:00.655590 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:00.655719 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:00.658997 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:00.659002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:00.659005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:00.659034 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:00.661263 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:00.661948 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:00.661953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:00.661957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:00.662133 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:00.662529 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:00.662533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:00.662536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:00.662785 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:00.662937 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:00.663518 (+ 581us) server_negotiation.cc:299] Negotiation successful
0927 06:55:00.663709 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 06:55:00.664991 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:00.648144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.3:46099)
0927 06:55:00.648798 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:00.648837 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:00.650524 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:00.651485 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:00.651496 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:00.652063 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:00.652689 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:00.652706 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:00.653816 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:00.653821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:00.655056 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:00.655067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:00.655800 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:00.655809 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:00.656047 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:00.656637 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:00.656668 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:00.658793 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:00.661404 (+ 2611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:00.661419 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:00.661433 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:00.661808 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:00.662235 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:00.662240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:00.662244 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:00.662416 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:00.662904 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:00.662911 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:00.663195 (+ 284us) client_negotiation.cc:769] Sending connection context
0927 06:55:00.663436 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 06:55:00.663878 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":263,"threads_started":1}
I20260927 06:55:00.666319 7007 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44789
I20260927 06:55:00.666560 7007 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:00.667153 7007 heartbeater.cc:511] Master 127.0.97.62:44789 requested a full tablet report, sending...
I20260927 06:55:00.668365 6551 ts_manager.cc:194] Registered new tserver with Master: 9eebd909c11f4326bb832093502f823f (127.0.97.3:41593)
I20260927 06:55:00.669123 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46099
I20260927 06:55:00.677322 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260927 06:55:00.722703 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33033
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44789,127.0.97.61:33033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:55:00.725917 6740 heartbeater.cc:503] Master 127.0.97.62:44789 was elected leader, sending a full tablet report...
W20260927 06:55:01.035843 7012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:01.036120 7012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:01.036336 7012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:01.043815 7012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:55:01.043917 7012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:01.044025 7012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:01.044116 7012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:55:01.044193 7012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:55:01.054837 7012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44789,127.0.97.61:33033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
W20260927 06:55:01.055197 7012 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:44789,127.0.97.61:33033'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 06:55:01.056224 7012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:01.057576 7012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:01.068637 7018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:01.068693 7020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:01.069000 7017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:01.069818 7012 server_base.cc:1034] running on GCE node
I20260927 06:55:01.070605 7012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:01.071838 7012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:01.073040 7012 hybrid_clock.cc:648] HybridClock initialized: now 1790492101072976 us; error 36 us; skew 500 ppm
I20260927 06:55:01.073352 7012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:01 dist-test-slave-f892 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492101, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:01.076354 7012 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260927 06:55:01.077752 7012 webserver.cc:460] Webserver started at http://127.0.97.61:34735/ using document root <none> and password file <none>
I20260927 06:55:01.078478 7012 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:01.078562 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:01.078833 7012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:01.081627 7012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/data/instance:
uuid: "9d32914ce3024b72a55b8703f765deba"
format_stamp: "Formatted at 2026-09-27 06:55:01 on dist-test-slave-f892"
server_key: "e48ced0af18a8864310668e6e278b8b7"
server_key_iv: "b827516e8ef8b4be7cf9f32325d4f1ac"
server_key_version: "encryptionkey@0"
I20260927 06:55:01.082410 7012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/wal/instance:
uuid: "9d32914ce3024b72a55b8703f765deba"
format_stamp: "Formatted at 2026-09-27 06:55:01 on dist-test-slave-f892"
server_key: "e48ced0af18a8864310668e6e278b8b7"
server_key_iv: "b827516e8ef8b4be7cf9f32325d4f1ac"
server_key_version: "encryptionkey@0"
I20260927 06:55:01.087471 7012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260927 06:55:01.090731 7027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:01.092104 7012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 06:55:01.092252 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/wal
uuid: "9d32914ce3024b72a55b8703f765deba"
format_stamp: "Formatted at 2026-09-27 06:55:01 on dist-test-slave-f892"
server_key: "e48ced0af18a8864310668e6e278b8b7"
server_key_iv: "b827516e8ef8b4be7cf9f32325d4f1ac"
server_key_version: "encryptionkey@0"
I20260927 06:55:01.092415 7012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:01.134378 7012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:01.138213 7012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:01.138428 7012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:01.155995 7012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:33033
I20260927 06:55:01.156008 7078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:33033 every 8 connection(s)
I20260927 06:55:01.157397 7012 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/data/info.pb
I20260927 06:55:01.161298 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:01.161306 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 7012
I20260927 06:55:01.170463 7079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 06:55:01 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 06:55:01 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492101, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 06:55:01.189714 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.173917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34098 (local address 127.0.97.62:44789)
0927 06:55:01.174382 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:01.174390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:01.174917 (+ 527us) server_negotiation.cc:407] Connection header received
0927 06:55:01.175121 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:01.175136 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:01.175195 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:01.175302 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:01.176588 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.177479 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.179325 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.179690 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.179787 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.183484 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:01.183491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:01.183497 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:01.183539 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:01.186689 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.187557 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.187571 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.187574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.187704 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.188044 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.188049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.188052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.188471 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:01.188617 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:01.189157 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 06:55:01.189354 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":239,"threads_started":1}
I20260927 06:55:01.190155 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.163589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.0.97.61:33033)
0927 06:55:01.169656 (+ 6067us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:01.169668 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:01.169707 (+ 39us) server_negotiation.cc:407] Connection header received
0927 06:55:01.171266 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:01.171296 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:01.171657 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:01.171979 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:01.173736 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.175576 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.177085 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.177949 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.178139 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.182833 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:01.182840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:01.182851 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:01.182881 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:01.186199 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.186791 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.186798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.186807 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.186973 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.187420 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.187423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.187425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.188046 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:01.188317 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:01.188665 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 06:55:01.189049 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5842,"thread_start_us":5604,"threads_started":1}
I20260927 06:55:01.190135 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.173973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.0.1:34098)
0927 06:55:01.174797 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:01.174835 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:01.174989 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:01.175360 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:01.175366 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:01.176116 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:01.176418 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.176429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.177621 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.177625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.179181 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.179195 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.179916 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.179925 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.180127 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.180750 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:01.180789 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:01.183232 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:01.186831 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.186839 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.186844 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.187432 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.187805 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.187807 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.187809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.187931 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.188631 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:01.188639 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:01.188811 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 06:55:01.189048 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 06:55:01.189332 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":336,"threads_started":1}
I20260927 06:55:01.191883 7079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } has no permanent_uuid. Determining permanent_uuid...
I20260927 06:55:01.193222 6874 heartbeater.cc:503] Master 127.0.97.62:44789 was elected leader, sending a full tablet report...
Sep 27 06:55:01 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492101, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 06:55:01.203583 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.192679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33033 (local address 127.0.0.1:32806)
0927 06:55:01.192886 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:01.192903 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:01.193170 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:01.193471 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:01.193474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:01.193853 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:01.193976 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.193984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.194998 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.195006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.195823 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.195832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.196517 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.196523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.196712 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.197256 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:01.197272 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:01.199229 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:01.201712 (+ 2483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.201716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.201718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.202115 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.202424 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.202426 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.202429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.202537 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.202971 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:01.202974 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:01.203088 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 06:55:01.203185 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 06:55:01.203363 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 06:55:01.203715 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.193064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32806 (local address 127.0.97.61:33033)
0927 06:55:01.193203 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:01.193212 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:01.193230 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:55:01.193273 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:01.193278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:01.193351 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:01.193440 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:01.194121 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.194881 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.195953 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.196390 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.196513 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.199398 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:01.199403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:01.199407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:01.199432 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:01.201605 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.202225 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.202228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.202231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.202339 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.202624 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.202627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.202628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.202874 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:01.202969 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:01.203267 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 06:55:01.203459 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":48}
I20260927 06:55:01.208168 7079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: Bootstrap starting.
I20260927 06:55:01.211012 7079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:01.211979 7079 log.cc:824] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: Log is configured to *not* fsync() on all Append() calls
I20260927 06:55:01.214391 7079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: No bootstrap required, opened a new log
I20260927 06:55:01.220942 7079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } }
I20260927 06:55:01.221251 7079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:01.221350 7079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d32914ce3024b72a55b8703f765deba, State: Initialized, Role: FOLLOWER
I20260927 06:55:01.221853 7079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } }
I20260927 06:55:01.222625 7086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } } }
I20260927 06:55:01.222882 7086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:55:01.223503 7079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: configured and running, proceeding with master startup.
W20260927 06:55:01.231696 7097 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 06:55:01.231830 7097 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 06:55:01.233973 7012 master_runner.cc:428] Detected that this master 9d32914ce3024b72a55b8703f765deba is joining an existing cluster
I20260927 06:55:01.234114 7012 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:33033
I20260927 06:55:01.254444 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.238318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33033 (local address 127.0.0.1:32812)
0927 06:55:01.239155 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:01.239172 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:01.239291 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:01.239611 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:01.239616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:01.240464 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:01.240842 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.240854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.242345 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.242348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.243586 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.243604 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.244731 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.244737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.244904 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.245629 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:01.245653 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:01.247022 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:01.251240 (+ 4218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.251248 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.251254 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.251763 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.252162 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.252168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.252173 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.252310 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.253075 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:01.253081 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:01.253300 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 06:55:01.253482 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 06:55:01.253762 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":1,"spinlock_wait_cycles":23424,"thread_start_us":509,"threads_started":1}
I20260927 06:55:01.254984 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.237757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.0.97.62:44789)
0927 06:55:01.238167 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:01.238174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:01.238930 (+ 756us) server_negotiation.cc:407] Connection header received
0927 06:55:01.239207 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:01.239212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:01.239301 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:01.239420 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:01.240922 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.242141 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.244236 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.244824 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.244933 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.247951 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:01.247957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:01.247961 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:01.248261 (+ 300us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:01.251914 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.252695 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.252701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.252704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.252849 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.253308 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.253313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.253316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.253658 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:01.253774 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:01.254397 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 06:55:01.254643 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284}
I20260927 06:55:01.255324 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.237927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32812 (local address 127.0.97.61:33033)
0927 06:55:01.238161 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:01.238170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:01.239235 (+ 1065us) server_negotiation.cc:407] Connection header received
0927 06:55:01.239409 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:01.239413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:01.239478 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:01.239567 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:01.241035 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.242144 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.243723 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.244357 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.244452 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.247233 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:01.247242 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:01.247249 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:01.247289 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:01.251098 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.251882 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.251889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.251892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.252016 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.252429 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.252436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.252441 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.252839 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:01.253018 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:01.254741 (+ 1723us) server_negotiation.cc:299] Negotiation successful
0927 06:55:01.254939 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 06:55:01.255985 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.237810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.0.1:34114)
0927 06:55:01.238801 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:01.238827 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:01.239051 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:01.239611 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:01.239615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:01.240399 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:01.240766 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.240776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.242275 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.242281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.244055 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.244080 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.245541 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.245551 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.245765 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.246404 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:01.246427 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:01.247729 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:01.252088 (+ 4359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.252093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.252096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.252540 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.253012 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.253017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.253020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.253174 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.253867 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:01.253872 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:01.254088 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 06:55:01.255220 (+ 1132us) client_negotiation.cc:240] Negotiation successful
0927 06:55:01.255528 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":26,"thread_start_us":564,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 27 06:55:01 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492099, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 06:55:01.279577 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.261982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32820 (local address 127.0.97.61:33033)
0927 06:55:01.262181 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:01.262187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:01.262829 (+ 642us) server_negotiation.cc:407] Connection header received
0927 06:55:01.263186 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:01.263192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:01.263273 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:01.263380 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:01.264763 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.265982 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.267851 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.268666 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.268792 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.272723 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:01.272732 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:01.272741 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:01.272776 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:01.276342 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.277208 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.277213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.277217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.277364 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.277801 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.277806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.277810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.278201 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:01.278345 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:01.278918 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 06:55:01.279147 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260927 06:55:01.279735 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.262053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33033 (local address 127.0.0.1:32820)
0927 06:55:01.262712 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:01.262755 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:01.262961 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:01.263454 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:01.263462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:01.264292 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:01.264615 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.264626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.266143 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.266148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.267676 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.267692 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.268833 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.268839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.269055 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.269695 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:01.269719 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:01.272539 (+ 2820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:01.276486 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.276498 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.276509 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.277071 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.277503 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.277511 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.277518 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.277682 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.278344 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:01.278356 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:01.278570 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 06:55:01.278788 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 06:55:01.279159 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":282,"threads_started":1}
I20260927 06:55:01.282030 6551 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:33033: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9d32914ce3024b72a55b8703f765deba"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 33033
}
attrs {
promote: true
}
}
dest_uuid: "f346c97981314f6c97c333fb51cf4e55"
cas_config_opid_index: -1
I20260927 06:55:01.283207 6551 consensus_queue.cc:237] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true } }
I20260927 06:55:01.292656 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9d32914ce3024b72a55b8703f765deba (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true } } }
I20260927 06:55:01.293260 7054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:01.297379 7054 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 1 FOLLOWER]: Refusing update from remote peer f346c97981314f6c97c333fb51cf4e55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 06:55:01.297474 7109 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:33033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260927 06:55:01.298799 7111 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:55:01.299335 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38017
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44789,127.0.97.61:33033,127.0.97.60:38017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:55:01.305665 7012 master.cc:537] Master@127.0.97.61:33033 shutting down...
I20260927 06:55:01.305546 7110 sys_catalog.cc:455] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true } } }
I20260927 06:55:01.305825 7110 sys_catalog.cc:458] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:01.320560 7054 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9d32914ce3024b72a55b8703f765deba (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true } } }
I20260927 06:55:01.333369 7116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true } } }
I20260927 06:55:01.333623 7116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: This master's current role is: LEARNER
I20260927 06:55:01.334048 7086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: SysCatalogTable state changed. Reason: New leader f346c97981314f6c97c333fb51cf4e55. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true } } }
I20260927 06:55:01.334283 7086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: This master's current role is: LEARNER
I20260927 06:55:01.334118 7119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true } } }
I20260927 06:55:01.334470 7119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: This master's current role is: LEARNER
I20260927 06:55:01.352164 7012 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 1 LEARNER]: Raft consensus shutting down.
I20260927 06:55:01.352717 7012 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 1 LEARNER]: Raft consensus is shut down!
I20260927 06:55:01.352901 7012 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: stopping tablet replica
I20260927 06:55:01.370510 7012 master.cc:552] Master@127.0.97.61:33033 shutdown complete.
I20260927 06:55:01.370620 7012 master_runner.cc:305] Clearing existing system tablet
I20260927 06:55:01.371062 7012 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:01.371158 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:01.374683 7012 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 06:55:01.376369 7121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:01.376718 7012 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:01.377000 7012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 06:55:01.377085 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/wal
uuid: "9d32914ce3024b72a55b8703f765deba"
format_stamp: "Formatted at 2026-09-27 06:55:01 on dist-test-slave-f892"
server_key: "e48ced0af18a8864310668e6e278b8b7"
server_key_iv: "b827516e8ef8b4be7cf9f32325d4f1ac"
server_key_version: "encryptionkey@0"
I20260927 06:55:01.381394 7012 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:01.388293 7012 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 06:55:01.388448 7012 log.cc:1196] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 06:55:01.388933 7012 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: Deleting consensus metadata
I20260927 06:55:01.389183 7012 master_runner.cc:315] Copying system tablet from 127.0.97.62:44789
I20260927 06:55:01.391697 7012 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:44789
I20260927 06:55:01.408315 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.392541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.0.97.62:44789)
0927 06:55:01.392736 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:01.392742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:01.393231 (+ 489us) server_negotiation.cc:407] Connection header received
0927 06:55:01.393436 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:01.393443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:01.393537 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:01.393641 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:01.394744 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.395876 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.397917 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.398748 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.398870 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.401571 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:01.401578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:01.401584 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:01.401624 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:01.405229 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.406079 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.406085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.406089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.406235 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.406641 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.406646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.406650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.407007 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:01.407145 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:01.407711 (+ 566us) server_negotiation.cc:299] Negotiation successful
0927 06:55:01.407934 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 06:55:01.408609 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.392462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.0.1:34120)
0927 06:55:01.393126 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:01.393146 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:01.393294 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:01.393687 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:01.393692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:01.394265 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:01.394567 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.394584 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.396042 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.396049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.397699 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.397718 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.399231 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.399241 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.399499 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.400258 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:01.400284 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:01.401374 (+ 1090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:01.405382 (+ 4008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.405387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.405390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.405869 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.406362 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.406367 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.406371 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.406527 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.407163 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:01.407169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:01.407379 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 06:55:01.407596 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 06:55:01.407954 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":418,"threads_started":1}
I20260927 06:55:01.409511 6571 tablet_copy_service.cc:140] P f346c97981314f6c97c333fb51cf4e55: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9d32914ce3024b72a55b8703f765deba ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:34120)
I20260927 06:55:01.409768 6571 tablet_copy_service.cc:161] P f346c97981314f6c97c333fb51cf4e55: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9d32914ce3024b72a55b8703f765deba at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:34120: session id = 9d32914ce3024b72a55b8703f765deba-00000000000000000000000000000000
I20260927 06:55:01.414263 6571 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 06:55:01.416391 7012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 06:55:01.423213 7012 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: tablet copy: Starting download of 0 data blocks...
I20260927 06:55:01.423579 7012 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: tablet copy: Starting download of 1 WAL segments...
I20260927 06:55:01.426594 7012 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 06:55:01.430729 6571 tablet_copy_service.cc:342] P f346c97981314f6c97c333fb51cf4e55: Request end of tablet copy session 9d32914ce3024b72a55b8703f765deba-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:34120
I20260927 06:55:01.430939 6571 tablet_copy_service.cc:433] P f346c97981314f6c97c333fb51cf4e55: ending tablet copy session 9d32914ce3024b72a55b8703f765deba-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9d32914ce3024b72a55b8703f765deba
I20260927 06:55:01.443598 7012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:01.447593 7128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:01.448125 7129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:01.455518 7012 server_base.cc:1034] running on GCE node
W20260927 06:55:01.455513 7131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:01.456218 7012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:01.456840 7012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:01.458066 7012 hybrid_clock.cc:648] HybridClock initialized: now 1790492101457987 us; error 31 us; skew 500 ppm
I20260927 06:55:01.458299 7012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:01 dist-test-slave-f892 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492101, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:01.460592 7012 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260927 06:55:01.461783 7012 webserver.cc:460] Webserver started at http://127.0.97.61:44459/ using document root <none> and password file <none>
I20260927 06:55:01.462158 7012 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:01.462245 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:01.465157 7012 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260927 06:55:01.466964 7138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:01.467360 7012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 06:55:01.467487 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/wal
uuid: "9d32914ce3024b72a55b8703f765deba"
format_stamp: "Formatted at 2026-09-27 06:55:01 on dist-test-slave-f892"
server_key: "e48ced0af18a8864310668e6e278b8b7"
server_key_iv: "b827516e8ef8b4be7cf9f32325d4f1ac"
server_key_version: "encryptionkey@0"
I20260927 06:55:01.467638 7012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:01.492933 7012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:01.496735 7012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:01.524772 7012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:33033
I20260927 06:55:01.524959 7189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:33033 every 8 connection(s)
I20260927 06:55:01.525801 7012 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-1/data/info.pb
I20260927 06:55:01.530936 7190 sys_catalog.cc:263] Verifying existing consensus state
I20260927 06:55:01.535077 7190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: Bootstrap starting.
I20260927 06:55:01.550406 7190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 06:55:01.550881 7190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: Bootstrap complete.
I20260927 06:55:01.551817 7190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true } }
I20260927 06:55:01.552086 7190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d32914ce3024b72a55b8703f765deba, State: Initialized, Role: LEARNER
I20260927 06:55:01.552392 7190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true } }
I20260927 06:55:01.553157 7193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true } } }
I20260927 06:55:01.553553 7193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: This master's current role is: LEARNER
I20260927 06:55:01.553530 7190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:55:01.564031 7204 catalog_manager.cc:1259] Loaded cluster ID: 17f0f544b36d4aad874de996e288df85
I20260927 06:55:01.564149 7204 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: loading cluster ID for follower catalog manager: success
I20260927 06:55:01.567668 7204 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: acquiring CA information for follower catalog manager: success
I20260927 06:55:01.570050 7204 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 06:55:01.671463 7007 heartbeater.cc:503] Master 127.0.97.62:44789 was elected leader, sending a full tablet report...
W20260927 06:55:01.686952 7112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:01.687275 7112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:01.687433 7112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:01.694804 7112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:55:01.694933 7112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:01.695039 7112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:01.695139 7112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:55:01.695212 7112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:55:01.705087 7112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44789,127.0.97.61:33033,127.0.97.60:38017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:01.706354 7112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:01.707561 7112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:01.719050 7209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:01.719103 7210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:01.719100 7212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:01.727507 7112 server_base.cc:1034] running on GCE node
I20260927 06:55:01.728392 7112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:01.730043 7112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:01.731252 7112 hybrid_clock.cc:648] HybridClock initialized: now 1790492101731218 us; error 28 us; skew 500 ppm
I20260927 06:55:01.731590 7112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:01 dist-test-slave-f892 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492101, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:01.734472 7112 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260927 06:55:01.735880 7112 webserver.cc:460] Webserver started at http://127.0.97.60:35213/ using document root <none> and password file <none>
I20260927 06:55:01.736560 7112 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:01.736692 7112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:01.736971 7112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:01.739800 7112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/data/instance:
uuid: "30e8b6e93caf4f649f2c446ede4d7957"
format_stamp: "Formatted at 2026-09-27 06:55:01 on dist-test-slave-f892"
server_key: "3b1659d4a6018391190b225a03b7e49d"
server_key_iv: "aa92c8cacb129373d9fbfa52cf8a3f51"
server_key_version: "encryptionkey@0"
I20260927 06:55:01.740506 7112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/wal/instance:
uuid: "30e8b6e93caf4f649f2c446ede4d7957"
format_stamp: "Formatted at 2026-09-27 06:55:01 on dist-test-slave-f892"
server_key: "3b1659d4a6018391190b225a03b7e49d"
server_key_iv: "aa92c8cacb129373d9fbfa52cf8a3f51"
server_key_version: "encryptionkey@0"
I20260927 06:55:01.745183 7112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 06:55:01.748911 7219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:01.750232 7112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:55:01.750396 7112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/wal
uuid: "30e8b6e93caf4f649f2c446ede4d7957"
format_stamp: "Formatted at 2026-09-27 06:55:01 on dist-test-slave-f892"
server_key: "3b1659d4a6018391190b225a03b7e49d"
server_key_iv: "aa92c8cacb129373d9fbfa52cf8a3f51"
server_key_version: "encryptionkey@0"
I20260927 06:55:01.750563 7112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:01.772074 7112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:01.775666 7112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:01.775931 7112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:01.794667 7112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:38017
I20260927 06:55:01.794698 7270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:38017 every 8 connection(s)
I20260927 06:55:01.795866 7112 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/data/info.pb
I20260927 06:55:01.799273 7271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:01.799870 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 7112
I20260927 06:55:01.807860 7271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 06:55:01 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 06:55:01 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492101, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 06:55:01.823484 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.801727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38118 (local address 127.0.97.60:38017)
0927 06:55:01.803104 (+ 1377us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:01.803114 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:01.803160 (+ 46us) server_negotiation.cc:407] Connection header received
0927 06:55:01.805535 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:01.805554 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:01.805895 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:01.806319 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:01.807711 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.809146 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.810877 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.811847 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.812034 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.816263 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:01.816271 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:01.816280 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:01.816311 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:01.819706 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.820487 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.820494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.820499 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.820642 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.821029 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.821038 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.821040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.821468 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:01.821751 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:01.822220 (+ 469us) server_negotiation.cc:299] Negotiation successful
0927 06:55:01.822592 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1210,"thread_start_us":451,"threads_started":1}
I20260927 06:55:01.824702 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.810354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34122 (local address 127.0.97.62:44789)
0927 06:55:01.810547 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:01.810555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:01.811094 (+ 539us) server_negotiation.cc:407] Connection header received
0927 06:55:01.811249 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:01.811253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:01.811313 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:01.811389 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:01.812669 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.813323 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.814683 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.815576 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.815702 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.819504 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:01.819509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:01.819512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:01.819541 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:01.821910 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.822749 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.822754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.822759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.822896 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.823281 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.823288 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.823296 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.823526 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:01.823697 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:01.824163 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 06:55:01.824362 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 06:55:01.824865 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.810265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.0.1:34122)
0927 06:55:01.810943 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:01.810980 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:01.811130 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:01.811474 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:01.811481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:01.812299 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:01.812524 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.812532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.813454 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.813463 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.814527 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.814537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.815788 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.815794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.815933 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.816542 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:01.816559 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:01.819326 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:01.822079 (+ 2753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.822089 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.822096 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.822638 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.823010 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.823014 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.823018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.823154 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.823731 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:01.823738 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:01.823879 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 06:55:01.824061 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 06:55:01.824354 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":339,"threads_started":1}
I20260927 06:55:01.826264 7271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 06:55:01 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492101, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 06:55:01.838325 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.826966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33033 (local address 127.0.0.1:32824)
0927 06:55:01.827144 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:01.827160 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:01.827242 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:01.827718 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:01.827721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:01.827968 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:01.828083 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.828090 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.829341 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.829344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.830398 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.830408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.831007 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.831010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.831137 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.832009 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:01.832029 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:01.833802 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:01.836446 (+ 2644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.836450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.836452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.836761 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.837057 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.837060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.837062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.837167 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.837636 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:01.837639 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:01.837744 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 06:55:01.837865 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 06:55:01.838117 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 06:55:01.838446 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.827026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32824 (local address 127.0.97.61:33033)
0927 06:55:01.827493 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:01.827498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:01.827513 (+ 15us) server_negotiation.cc:407] Connection header received
0927 06:55:01.827556 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:01.827559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:01.827609 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:01.827691 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:01.828230 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.829232 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.830512 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.830896 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.830996 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.833974 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:01.833978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:01.833981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:01.834045 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:01.836335 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.836847 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.836850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.836853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.836970 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.837246 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.837249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.837251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.837536 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:01.837648 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:01.837944 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 06:55:01.838169 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":271,"threads_started":1}
I20260927 06:55:01.839538 7271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 06:55:01 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492101, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 06:55:01.848131 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true } } }
I20260927 06:55:01.848484 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:01.852403 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.840176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38017 (local address 127.0.0.1:38130)
0927 06:55:01.840359 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:01.840388 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:01.840458 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:01.840685 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:01.840689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:01.840976 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:01.841110 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.841117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.842117 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.842121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.842899 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.842908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.843499 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.843503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.843651 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.844234 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:01.844249 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:01.845991 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:01.849394 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.849413 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.849420 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.850007 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.850473 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.850480 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.850485 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.850635 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.851249 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:01.851256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:01.851451 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 06:55:01.851640 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 06:55:01.851988 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 06:55:01.852816 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.840259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38130 (local address 127.0.97.60:38017)
0927 06:55:01.840432 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:01.840436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:01.840451 (+ 15us) server_negotiation.cc:407] Connection header received
0927 06:55:01.840528 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:01.840531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:01.840579 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:01.840650 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:01.841220 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.841956 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.843021 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.843377 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.843474 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.846219 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:01.846224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:01.846227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:01.846251 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:01.849232 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.850199 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.850206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.850212 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.850335 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.850747 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.850752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.850754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.851087 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:01.851244 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:01.851745 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 06:55:01.852521 (+ 776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":536,"server-negotiator.queue_time_us":78}
I20260927 06:55:01.856884 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.848534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32832 (local address 127.0.97.61:33033)
0927 06:55:01.850797 (+ 2263us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:01.850805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:01.850879 (+ 74us) server_negotiation.cc:407] Connection header received
0927 06:55:01.851280 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:01.851285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:01.851375 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:01.851439 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:01.852015 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.853065 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.855137 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.855997 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.856105 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.856331 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:01.856549 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 06:55:01.856686 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1640}
I20260927 06:55:01.857021 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.848239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33033 (local address 127.0.0.1:32832)
0927 06:55:01.850777 (+ 2538us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:01.850804 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:01.851121 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:01.851585 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:01.851588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:01.851644 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:01.851856 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.851868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.853194 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.853201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.854965 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.854981 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.856173 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.856178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.856375 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.856429 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 06:55:01.856523 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 06:55:01.856654 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2082,"thread_start_us":1757,"threads_started":1}
I20260927 06:55:01.857839 7193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: SysCatalogTable state changed. Reason: New leader f346c97981314f6c97c333fb51cf4e55. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true } } }
I20260927 06:55:01.858062 7271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: Bootstrap starting.
I20260927 06:55:01.858119 7193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: This master's current role is: LEARNER
I20260927 06:55:01.858525 7277 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55: attempting to promote NON_VOTER 9d32914ce3024b72a55b8703f765deba to VOTER
I20260927 06:55:01.859218 7277 consensus_queue.cc:237] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } }
I20260927 06:55:01.860818 7165 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 1 LEARNER]: Refusing update from remote peer f346c97981314f6c97c333fb51cf4e55: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 06:55:01.861328 7279 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 06:55:01.861435 7271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:01.862551 7271 log.cc:824] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: Log is configured to *not* fsync() on all Append() calls
I20260927 06:55:01.864548 7277 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9d32914ce3024b72a55b8703f765deba (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } }
I20260927 06:55:01.865381 7271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: No bootstrap required, opened a new log
I20260927 06:55:01.865218 7165 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9d32914ce3024b72a55b8703f765deba (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } }
I20260927 06:55:01.867439 7279 sys_catalog.cc:455] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } }
I20260927 06:55:01.867390 7278 sys_catalog.cc:455] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } }
I20260927 06:55:01.867723 7278 sys_catalog.cc:458] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:01.867705 7279 sys_catalog.cc:458] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:01.868782 7193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } }
I20260927 06:55:01.869001 7193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:55:01.874230 7271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } }
I20260927 06:55:01.874514 7271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:01.874596 7271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e8b6e93caf4f649f2c446ede4d7957, State: Initialized, Role: FOLLOWER
I20260927 06:55:01.875125 7271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } }
I20260927 06:55:01.875841 7284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } } }
I20260927 06:55:01.876185 7284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:55:01.876868 7271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 06:55:01.885447 7295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 06:55:01.885576 7295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 06:55:01.889334 7112 master_runner.cc:428] Detected that this master 30e8b6e93caf4f649f2c446ede4d7957 is joining an existing cluster
I20260927 06:55:01.889428 7112 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:38017
I20260927 06:55:01.911185 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.894616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.0.97.61:33033)
0927 06:55:01.894790 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:01.894796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:01.895408 (+ 612us) server_negotiation.cc:407] Connection header received
0927 06:55:01.895640 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:01.895645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:01.895716 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:01.895780 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:01.897166 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.898416 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.900864 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.901510 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.901605 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.904345 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:01.904354 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:01.904360 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:01.904396 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:01.907620 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.908351 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.908356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.908360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.908497 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.909479 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.909485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.909488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.909810 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:01.909944 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:01.910600 (+ 656us) server_negotiation.cc:299] Negotiation successful
0927 06:55:01.910803 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 06:55:01.911489 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.893475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34130 (local address 127.0.97.62:44789)
0927 06:55:01.893694 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:01.893700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:01.894329 (+ 629us) server_negotiation.cc:407] Connection header received
0927 06:55:01.894595 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:01.894600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:01.894676 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:01.894755 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:01.896389 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.897881 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.901156 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.901987 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.902117 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.904657 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:01.904664 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:01.904669 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:01.904705 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:01.908461 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.909170 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.909177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.909182 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.909324 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.909832 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.909840 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.909845 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.910196 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:01.910319 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:01.910926 (+ 607us) server_negotiation.cc:299] Negotiation successful
0927 06:55:01.911148 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 06:55:01.912756 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.893529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.0.1:34130)
0927 06:55:01.894209 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:01.894231 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:01.894438 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:01.895025 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:01.895030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:01.895732 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:01.896177 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.896193 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.898130 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.898139 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.900958 (+ 2819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.900977 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.902398 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.902404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.902596 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.903255 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:01.903283 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:01.904467 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:01.908610 (+ 4143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.908614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.908617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.909035 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.909471 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.909478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.909483 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.909697 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.910408 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:01.910413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:01.910612 (+ 199us) client_negotiation.cc:769] Sending connection context
0927 06:55:01.911818 (+ 1206us) client_negotiation.cc:240] Negotiation successful
0927 06:55:01.912200 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":27,"thread_start_us":358,"threads_started":1}
I20260927 06:55:01.914106 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.894641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33033 (local address 127.0.0.1:32846)
0927 06:55:01.895308 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:01.895328 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:01.895501 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:01.895986 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:01.895993 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:01.896676 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:01.897003 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.897014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.898629 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.898638 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.900700 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.900714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.901892 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.901898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.902125 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.902829 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:01.902851 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:01.904144 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:01.907756 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.907761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.907765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.908217 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.909051 (+ 834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.909057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.909066 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.909345 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.910062 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:01.910067 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:01.910287 (+ 220us) client_negotiation.cc:769] Sending connection context
0927 06:55:01.911715 (+ 1428us) client_negotiation.cc:240] Negotiation successful
0927 06:55:01.912047 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":3,"thread_start_us":353,"threads_started":1}
I20260927 06:55:01.917192 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.895452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.0.97.60:38017)
0927 06:55:01.895615 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:01.895621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:01.902348 (+ 6727us) server_negotiation.cc:407] Connection header received
0927 06:55:01.902651 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:01.902655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:01.902716 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:01.902791 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:01.904353 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.905408 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.907400 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.908020 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.908103 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.910974 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:01.910979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:01.910983 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:01.911012 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:01.914357 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.915011 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.915016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.915019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.915137 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.915452 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.915456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.915459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.915904 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:01.916119 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:01.916582 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 06:55:01.916813 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 06:55:01.917932 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.895716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38017 (local address 127.0.0.1:38134)
0927 06:55:01.902265 (+ 6549us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:01.902284 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:01.902543 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:01.902976 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:01.902982 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:01.903872 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:01.904209 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.904219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.905648 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.905656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.907253 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.907265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.908417 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.908423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.908605 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.909717 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:01.909773 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:01.910830 (+ 1057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:01.914483 (+ 3653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.914487 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.914492 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.914889 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.915246 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.915250 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.915254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.915354 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.916165 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:01.916169 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:01.916311 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 06:55:01.917397 (+ 1086us) client_negotiation.cc:240] Negotiation successful
0927 06:55:01.917630 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6404,"thread_start_us":6657,"threads_started":1}
Sep 27 06:55:01 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492099, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 06:55:01.928913 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 6612
I20260927 06:55:01.933455 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.919624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38017 (local address 127.0.0.1:38136)
0927 06:55:01.919803 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:01.919822 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:01.919921 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:01.920407 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:01.920410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:01.920734 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:01.920868 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.920875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.922274 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.922278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.923124 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:01.923134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.924057 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.924060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:01.924167 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.924695 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:01.924710 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:01.926600 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:01.930712 (+ 4112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.930718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.930722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.931243 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.931661 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:01.931666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:01.931670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:01.931822 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:01.932469 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:01.932474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:01.932656 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 06:55:01.932837 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 06:55:01.933090 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 06:55:01.933465 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.919916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.0.97.60:38017)
0927 06:55:01.920076 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:01.920082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:01.920100 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:55:01.920187 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:01.920192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:01.920279 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:01.920391 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:01.920997 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.922152 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.923255 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.923928 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.924060 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.926776 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:01.926783 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:01.926788 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:01.926822 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:01.930564 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.931398 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.931403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.931406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.931533 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.931979 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.931984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.931987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.932336 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:01.932486 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:01.932934 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 06:55:01.933143 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 06:55:01.935025 6551 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:38017: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 38017
}
attrs {
promote: true
}
}
dest_uuid: "f346c97981314f6c97c333fb51cf4e55"
cas_config_opid_index: 6
I20260927 06:55:01.935774 6551 consensus_queue.cc:237] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: true } }
I20260927 06:55:01.937973 7165 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 1 FOLLOWER]: Refusing update from remote peer f346c97981314f6c97c333fb51cf4e55: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 06:55:01.938180 7246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 06:55:01 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260927 06:55:01.942176 7277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 06:55:01.943284 7246 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 1 FOLLOWER]: Refusing update from remote peer f346c97981314f6c97c333fb51cf4e55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 06:55:01.944072 7277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:55:01.944813 7278 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 30e8b6e93caf4f649f2c446ede4d7957 (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: true } } }
I20260927 06:55:01.945379 7165 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 30e8b6e93caf4f649f2c446ede4d7957 (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: true } } }
I20260927 06:55:01.948966 7193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: true } } }
I20260927 06:55:01.949244 7193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:55:01.949242 7278 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:38017
I20260927 06:55:01.949518 7278 sys_catalog.cc:455] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: true } } }
I20260927 06:55:01.949771 7278 sys_catalog.cc:458] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:01.950450 7112 master.cc:537] Master@127.0.97.60:38017 shutting down...
I20260927 06:55:01.950273 7279 sys_catalog.cc:455] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: true } } }
I20260927 06:55:01.950546 7279 sys_catalog.cc:458] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:01.955874 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:01.929835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.0.97.1:36787)
0927 06:55:01.931014 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:01.931021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:01.931044 (+ 23us) server_negotiation.cc:407] Connection header received
0927 06:55:01.931094 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:01.931100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:01.931300 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:01.931475 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:01.932734 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.934382 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.935704 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:01.936352 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:01.936493 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:01.942950 (+ 6457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:01.942965 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:01.942982 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:01.943034 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:01.951762 (+ 8728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.952610 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.952620 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.952626 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.952764 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:01.953204 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:01.953208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:01.953211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:01.953657 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:01.955056 (+ 1399us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:01.955202 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 06:55:01.955483 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1025,"thread_start_us":873,"threads_started":1}
I20260927 06:55:01.957438 6735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:01.962729 7245 raft_consensus.cc:858] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } }
W20260927 06:55:01.963022 7245 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } }
I20260927 06:55:01.976092 7245 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9d32914ce3024b72a55b8703f765deba (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true } } }
I20260927 06:55:01.989312 7312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true } } }
I20260927 06:55:01.989714 7312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 06:55:01.990175 7284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f346c97981314f6c97c333fb51cf4e55. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true } } }
I20260927 06:55:01.990481 7284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 06:55:01.990202 7308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: true } } }
I20260927 06:55:01.990670 7308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 06:55:01.992270 7278 sys_catalog.cc:455] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: true } } }
I20260927 06:55:01.992552 7278 sys_catalog.cc:458] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:01.993357 7112 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260927 06:55:01.993786 7112 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
W20260927 06:55:01.993748 6538 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 06:55:01.993958 7112 tablet_replica.cc:330] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: stopping tablet replica
I20260927 06:55:02.004581 7112 master.cc:552] Master@127.0.97.60:38017 shutdown complete.
I20260927 06:55:02.004792 7112 master_runner.cc:305] Clearing existing system tablet
I20260927 06:55:02.005367 7112 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:02.005553 7112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:02.010706 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:02.010101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38017 (local address 127.0.0.1:38140)
0927 06:55:02.010303 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:02.010462 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 06:55:02.011236 6538 consensus_peers.cc:487] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 -> Peer 30e8b6e93caf4f649f2c446ede4d7957 (127.0.97.60:38017): Couldn't send request to peer 30e8b6e93caf4f649f2c446ede4d7957. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38017: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 06:55:02.014645 7112 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.008s sys 0.000s
I20260927 06:55:02.016345 7315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:02.016705 7112 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:02.016970 7112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260927 06:55:02.017077 7112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/wal
uuid: "30e8b6e93caf4f649f2c446ede4d7957"
format_stamp: "Formatted at 2026-09-27 06:55:01 on dist-test-slave-f892"
server_key: "3b1659d4a6018391190b225a03b7e49d"
server_key_iv: "aa92c8cacb129373d9fbfa52cf8a3f51"
server_key_version: "encryptionkey@0"
I20260927 06:55:02.020251 7112 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:02.024787 7112 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 06:55:02.024957 7112 log.cc:1196] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 06:55:02.025379 7112 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: Deleting consensus metadata
I20260927 06:55:02.025625 7112 master_runner.cc:315] Copying system tablet from 127.0.97.62:44789
I20260927 06:55:02.028180 7112 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:44789
I20260927 06:55:02.041145 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:02.028907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34146 (local address 127.0.97.62:44789)
0927 06:55:02.029162 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:02.029171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:02.029703 (+ 532us) server_negotiation.cc:407] Connection header received
0927 06:55:02.029919 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:02.029925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:02.030045 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:02.030143 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:02.030899 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.031601 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.033277 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.033686 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.033814 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:02.035663 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:02.035669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:02.035674 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:02.035714 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:02.038584 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:02.039217 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:02.039224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:02.039228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:02.039358 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:02.039736 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:02.039744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:02.039747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:02.040007 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:02.040165 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:02.040576 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 06:55:02.040780 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 06:55:02.041342 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:02.028892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.0.1:34146)
0927 06:55:02.029599 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:02.029614 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:02.029768 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:02.030166 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:02.030171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:02.030568 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:02.030758 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:02.030766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.031738 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.031744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:02.033125 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:02.033140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.033826 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.033836 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:02.034043 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:02.034745 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:02.034762 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:02.035465 (+ 703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:02.038714 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:02.038721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:02.038725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:02.039094 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:02.039469 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:02.039475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:02.039479 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:02.039627 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:02.040129 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:02.040134 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:02.040337 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 06:55:02.040480 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 06:55:02.040777 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":441,"threads_started":1}
I20260927 06:55:02.042164 6571 tablet_copy_service.cc:140] P f346c97981314f6c97c333fb51cf4e55: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 30e8b6e93caf4f649f2c446ede4d7957 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:34146)
I20260927 06:55:02.042331 6571 tablet_copy_service.cc:161] P f346c97981314f6c97c333fb51cf4e55: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 30e8b6e93caf4f649f2c446ede4d7957 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:34146: session id = 30e8b6e93caf4f649f2c446ede4d7957-00000000000000000000000000000000
I20260927 06:55:02.044783 6571 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 06:55:02.046334 7112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 06:55:02.051782 7112 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: tablet copy: Starting download of 0 data blocks...
I20260927 06:55:02.052047 7112 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: tablet copy: Starting download of 1 WAL segments...
I20260927 06:55:02.054049 7112 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 06:55:02.057327 6571 tablet_copy_service.cc:342] P f346c97981314f6c97c333fb51cf4e55: Request end of tablet copy session 30e8b6e93caf4f649f2c446ede4d7957-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:34146
I20260927 06:55:02.057458 6571 tablet_copy_service.cc:433] P f346c97981314f6c97c333fb51cf4e55: ending tablet copy session 30e8b6e93caf4f649f2c446ede4d7957-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 30e8b6e93caf4f649f2c446ede4d7957
I20260927 06:55:02.067734 7112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:02.070372 7322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:02.075582 7323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:02.080518 7112 server_base.cc:1034] running on GCE node
W20260927 06:55:02.080885 7325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:02.081259 7112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:02.081923 7112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:02.083130 7112 hybrid_clock.cc:648] HybridClock initialized: now 1790492102083093 us; error 28 us; skew 500 ppm
I20260927 06:55:02.083681 7112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:02 dist-test-slave-f892 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492102, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:02.085522 7112 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260927 06:55:02.086812 7112 webserver.cc:460] Webserver started at http://127.0.97.60:44807/ using document root <none> and password file <none>
I20260927 06:55:02.087096 7112 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:02.087204 7112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:02.089531 7112 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 06:55:02.091288 7332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:02.091715 7112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 06:55:02.091874 7112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/wal
uuid: "30e8b6e93caf4f649f2c446ede4d7957"
format_stamp: "Formatted at 2026-09-27 06:55:01 on dist-test-slave-f892"
server_key: "3b1659d4a6018391190b225a03b7e49d"
server_key_iv: "aa92c8cacb129373d9fbfa52cf8a3f51"
server_key_version: "encryptionkey@0"
I20260927 06:55:02.092082 7112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:02.110275 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 6612
I20260927 06:55:02.135080 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36787
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44789
--webserver_port=39717
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:44789,127.0.97.61:33033
--tserver_master_addrs=127.0.97.62:44789,127.0.97.61:33033,127.0.97.60:38017 with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:55:02.138677 7112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:02.143193 7112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:02.190836 7112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:38017
I20260927 06:55:02.190876 7384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:38017 every 8 connection(s)
I20260927 06:55:02.191797 7112 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/master-2/data/info.pb
I20260927 06:55:02.199321 7385 sys_catalog.cc:263] Verifying existing consensus state
I20260927 06:55:02.203608 7385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: Bootstrap starting.
I20260927 06:55:02.218438 7385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 06:55:02.218940 7385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: Bootstrap complete.
I20260927 06:55:02.219996 7385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: true } }
I20260927 06:55:02.220290 7385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 30e8b6e93caf4f649f2c446ede4d7957, State: Initialized, Role: LEARNER
I20260927 06:55:02.220568 7385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: true } }
I20260927 06:55:02.222477 7388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: true } } }
I20260927 06:55:02.222942 7385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:55:02.222949 7388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: This master's current role is: LEARNER
I20260927 06:55:02.244675 7399 catalog_manager.cc:1259] Loaded cluster ID: 17f0f544b36d4aad874de996e288df85
I20260927 06:55:02.244787 7399 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: loading cluster ID for follower catalog manager: success
I20260927 06:55:02.248185 7399 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: acquiring CA information for follower catalog manager: success
I20260927 06:55:02.250512 7399 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 06:55:02.484431 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: true } } }
I20260927 06:55:02.484757 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:02.490648 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:02.485131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.0.97.60:38017)
0927 06:55:02.485605 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:02.485609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:02.485624 (+ 15us) server_negotiation.cc:407] Connection header received
0927 06:55:02.485677 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:02.485680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:02.485741 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:02.485871 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:02.486276 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.487265 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.488842 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.489606 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.489756 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:02.490040 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:02.490215 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 06:55:02.490343 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":235,"threads_started":1}
I20260927 06:55:02.490638 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:02.484847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38017 (local address 127.0.0.1:38148)
0927 06:55:02.485042 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:02.485059 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:02.485157 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:02.485894 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:02.485898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:02.485924 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:02.486143 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:02.486152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.487517 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.487525 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:02.488688 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:02.488699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.489773 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.489779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:02.489996 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:02.490080 (+ 84us) client_negotiation.cc:769] Sending connection context
0927 06:55:02.490190 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 06:55:02.490330 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 06:55:02.491540 7360 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260927 06:55:02.491945 7388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f346c97981314f6c97c333fb51cf4e55. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: true } } }
I20260927 06:55:02.492213 7388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: This master's current role is: LEARNER
I20260927 06:55:02.492972 7278 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55: attempting to promote NON_VOTER 30e8b6e93caf4f649f2c446ede4d7957 to VOTER
I20260927 06:55:02.493685 7278 consensus_queue.cc:237] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: false } }
I20260927 06:55:02.495824 7360 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 1 LEARNER]: Refusing update from remote peer f346c97981314f6c97c333fb51cf4e55: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 06:55:02.496021 7165 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 1 FOLLOWER]: Refusing update from remote peer f346c97981314f6c97c333fb51cf4e55: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 06:55:02.496323 7279 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 06:55:02.496569 7277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 06:55:02.499007 7278 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 30e8b6e93caf4f649f2c446ede4d7957 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: false } } }
I20260927 06:55:02.499642 7165 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 30e8b6e93caf4f649f2c446ede4d7957 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: false } } }
I20260927 06:55:02.500422 7360 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 30e8b6e93caf4f649f2c446ede4d7957 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: false } } }
I20260927 06:55:02.503051 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: false } } }
I20260927 06:55:02.503253 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:02.503543 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: false } } }
I20260927 06:55:02.503778 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: This master's current role is: LEADER
W20260927 06:55:02.503681 7337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:02.504047 7337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:02.504184 7337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 06:55:02.504031 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: false } } }
I20260927 06:55:02.504036 7388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: false } } }
I20260927 06:55:02.504261 7388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:55:02.504271 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:02.504722 7405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f346c97981314f6c97c333fb51cf4e55" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: false } } }
I20260927 06:55:02.505009 7405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: This master's current role is: FOLLOWER
W20260927 06:55:02.512320 7337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:02.512410 7337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:02.512609 7337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:55:02.522467 7337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=39717
--tserver_master_addrs=127.0.97.62:44789,127.0.97.61:33033,127.0.97.60:38017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:02.523851 7337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:02.525166 7337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:02.539811 7408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:02.539960 7407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:02.539917 7410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:02.540709 7337 server_base.cc:1034] running on GCE node
I20260927 06:55:02.541348 7337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:02.542275 7337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:02.543493 7337 hybrid_clock.cc:648] HybridClock initialized: now 1790492102543456 us; error 50 us; skew 500 ppm
I20260927 06:55:02.543768 7337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:02 dist-test-slave-f892 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492102, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:02.547143 7337 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 06:55:02.548460 7337 webserver.cc:460] Webserver started at http://127.0.97.1:39717/ using document root <none> and password file <none>
I20260927 06:55:02.549067 7337 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:02.549173 7337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:02.553910 7337 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:55:02.556571 7417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:02.557960 7337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 06:55:02.558137 7337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "8b3b556cc4054b7ca08c3abe1a70ae40"
format_stamp: "Formatted at 2026-09-27 06:54:59 on dist-test-slave-f892"
server_key: "47b6de3af3a29eacf9174a379bdc540b"
server_key_iv: "c8c6356d2a6b26e6d69162993bce3b89"
server_key_version: "encryptionkey@0"
I20260927 06:55:02.559127 7337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:02.581424 7337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:02.584950 7337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:02.585150 7337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:02.585945 7337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:02.587169 7337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:02.587253 7337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:02.587327 7337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:02.587378 7337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:02.621861 7337 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36787
I20260927 06:55:02.621941 7529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36787 every 8 connection(s)
I20260927 06:55:02.623313 7337 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:55:02.630231 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 7337
I20260927 06:55:02.630558 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 6746
Sep 27 06:55:02 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492102, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 06:55:02 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492102, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 06:55:02 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492102, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 06:55:02.652907 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:02.628755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33033 (local address 127.0.97.1:41001)
0927 06:55:02.629322 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:02.629344 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:02.630729 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:02.631534 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:02.631545 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:02.632788 (+ 1243us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:02.633872 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:02.633895 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.635785 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.635790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:02.638469 (+ 2679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:02.638486 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.639825 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.639831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:02.640125 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:02.640985 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:02.641026 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:02.644086 (+ 3060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:02.648067 (+ 3981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:02.648077 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:02.648093 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:02.648798 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:02.649217 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:02.649223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:02.649226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:02.649462 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:02.650129 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:02.650137 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:02.650569 (+ 432us) client_negotiation.cc:769] Sending connection context
0927 06:55:02.650858 (+ 289us) client_negotiation.cc:240] Negotiation successful
0927 06:55:02.651361 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":373,"spinlock_wait_cycles":2944,"thread_start_us":455,"threads_started":1}
I20260927 06:55:02.654408 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:02.626715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39257 (local address 127.0.97.60:38017)
0927 06:55:02.626992 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:02.626998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:02.628368 (+ 1370us) server_negotiation.cc:407] Connection header received
0927 06:55:02.631022 (+ 2654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:02.631030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:02.631111 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:02.631190 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:02.634093 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.635574 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.638766 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.639415 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.639514 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:02.648333 (+ 8819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:02.648339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:02.648343 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:02.648383 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:02.651482 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:02.652305 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:02.652311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:02.652314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:02.652443 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:02.652880 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:02.652887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:02.652892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:02.653223 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:02.653353 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:02.653865 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 06:55:02.654103 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260927 06:55:02.654449 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:02.628917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41001 (local address 127.0.97.61:33033)
0927 06:55:02.629858 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:02.629862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:02.629877 (+ 15us) server_negotiation.cc:407] Connection header received
0927 06:55:02.631092 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:02.631097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:02.631181 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:02.631905 (+ 724us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:02.634580 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.635627 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.638874 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.639513 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.639687 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:02.644317 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:02.644324 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:02.644329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:02.644365 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:02.647908 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:02.648934 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:02.648939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:02.648942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:02.649098 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:02.649592 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:02.649597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:02.649601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:02.649940 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:02.653712 (+ 3772us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:02.653855 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 06:55:02.654082 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":705,"threads_started":1}
I20260927 06:55:02.655615 7532 heartbeater.cc:347] Connected to a master server at 127.0.97.61:33033
I20260927 06:55:02.655670 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:02.627258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38017 (local address 127.0.97.1:39257)
0927 06:55:02.628215 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:02.628267 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:02.630729 (+ 2462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:02.631534 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:02.631545 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:02.632788 (+ 1243us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:02.633872 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:02.633895 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.635785 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.635829 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:02.638620 (+ 2791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:02.638630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.640602 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.640608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:02.640782 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:02.641366 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:02.641390 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:02.648107 (+ 6717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:02.651636 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:02.651640 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:02.651643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:02.652151 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:02.652582 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:02.652587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:02.652593 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:02.652751 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:02.653432 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:02.653436 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:02.653571 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 06:55:02.654920 (+ 1349us) client_negotiation.cc:240] Negotiation successful
0927 06:55:02.655274 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":326,"spinlock_wait_cycles":36352,"thread_start_us":484,"threads_started":1}
I20260927 06:55:02.655944 7532 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:02.656592 7530 heartbeater.cc:347] Connected to a master server at 127.0.97.60:38017
I20260927 06:55:02.656725 7530 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:02.656924 7532 heartbeater.cc:511] Master 127.0.97.61:33033 requested a full tablet report, sending...
I20260927 06:55:02.657161 7530 heartbeater.cc:511] Master 127.0.97.60:38017 requested a full tablet report, sending...
I20260927 06:55:02.657724 7534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:02.626705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48539 (local address 127.0.97.62:44789)
0927 06:55:02.627419 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:02.627425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:02.628369 (+ 944us) server_negotiation.cc:407] Connection header received
0927 06:55:02.631030 (+ 2661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:02.631039 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:02.631124 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:02.631227 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:02.638869 (+ 7642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.640852 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.643920 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.644479 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.644604 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:02.649315 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:02.649321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:02.649326 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:02.649363 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:02.652685 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:02.653794 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:02.653799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:02.653802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:02.653952 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:02.654900 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:02.654907 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:02.654913 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:02.655950 (+ 1037us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:02.656143 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:02.656889 (+ 746us) server_negotiation.cc:299] Negotiation successful
0927 06:55:02.657166 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":502,"threads_started":1}
I20260927 06:55:02.658880 7535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:02.627258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.1:48539)
0927 06:55:02.628215 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:02.628265 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:02.630717 (+ 2452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:02.637669 (+ 6952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:02.637677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:02.638324 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:02.638723 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:02.638736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.642278 (+ 3542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.642284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:02.643783 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:02.643797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.644661 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.644669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:02.644862 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:02.645957 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:02.645981 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:02.649013 (+ 3032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:02.652837 (+ 3824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:02.652844 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:02.652848 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:02.653665 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:02.654167 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:02.654172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:02.654177 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:02.654380 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:02.656250 (+ 1870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:02.656256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:02.656412 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 06:55:02.658171 (+ 1759us) client_negotiation.cc:240] Negotiation successful
0927 06:55:02.658460 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":396,"threads_started":1}
I20260927 06:55:02.659229 7350 ts_manager.cc:194] Registered new tserver with Master: 8b3b556cc4054b7ca08c3abe1a70ae40 (127.0.97.1:36787)
Sep 27 06:55:02 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260927 06:55:02.659951 7155 ts_manager.cc:194] Registered new tserver with Master: 8b3b556cc4054b7ca08c3abe1a70ae40 (127.0.97.1:36787)
I20260927 06:55:02.660409 7531 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44789
I20260927 06:55:02.660537 7531 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:02.660907 7531 heartbeater.cc:511] Master 127.0.97.62:44789 requested a full tablet report, sending...
I20260927 06:55:02.661938 6551 ts_manager.cc:194] Re-registered known tserver with Master: 8b3b556cc4054b7ca08c3abe1a70ae40 (127.0.97.1:36787)
I20260927 06:55:02.662878 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48539
I20260927 06:55:02.666980 7540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:02.631852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.0.97.2:33493)
0927 06:55:02.640303 (+ 8451us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:02.640314 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:02.642087 (+ 1773us) server_negotiation.cc:407] Connection header received
0927 06:55:02.642305 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:02.642314 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:02.642507 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:02.642656 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:02.648531 (+ 5875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.651196 (+ 2665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.653151 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:02.653999 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:02.654167 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:02.660609 (+ 6442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:02.660622 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:02.660638 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:02.660678 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:02.663939 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:02.664701 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:02.664707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:02.664709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:02.664811 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:02.665181 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:02.665184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:02.665186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:02.665453 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:02.665646 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:02.665864 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 06:55:02.666105 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8231,"thread_start_us":9412,"threads_started":1}
I20260927 06:55:02.668208 6869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:02.749094 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 6746
I20260927 06:55:02.765341 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33493
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44789
--webserver_port=36849
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:44789,127.0.97.61:33033
--tserver_master_addrs=127.0.97.62:44789,127.0.97.61:33033,127.0.97.60:38017 with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:55:03.082753 7542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:03.083001 7542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:03.083173 7542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:03.090224 7542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:03.090351 7542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:03.090535 7542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:55:03.100227 7542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36849
--tserver_master_addrs=127.0.97.62:44789,127.0.97.61:33033,127.0.97.60:38017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:03.101387 7542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:03.102636 7542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:03.115393 7550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:03.115363 7548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:03.115962 7547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:03.116660 7542 server_base.cc:1034] running on GCE node
I20260927 06:55:03.117280 7542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:03.118242 7542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:03.119459 7542 hybrid_clock.cc:648] HybridClock initialized: now 1790492103119419 us; error 50 us; skew 500 ppm
I20260927 06:55:03.119760 7542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:03 dist-test-slave-f892 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:03.122897 7542 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 06:55:03.124197 7542 webserver.cc:460] Webserver started at http://127.0.97.2:36849/ using document root <none> and password file <none>
I20260927 06:55:03.124814 7542 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:03.124924 7542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:03.129786 7542 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 06:55:03.132225 7557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:03.133473 7542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 06:55:03.133594 7542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "151233c285ec4937b55164d138143479"
format_stamp: "Formatted at 2026-09-27 06:55:00 on dist-test-slave-f892"
server_key: "56223d1bbbf2aac48ad4ec4dd8557cc5"
server_key_iv: "0531d277826a4dd0d0e7c88d9162b7e3"
server_key_version: "encryptionkey@0"
I20260927 06:55:03.134531 7542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:03.150077 7542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:03.153592 7542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:03.153832 7542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:03.154817 7542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:03.156080 7542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:03.156183 7542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:03.156307 7542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:03.156412 7542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:03.187691 7542 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33493
I20260927 06:55:03.187748 7669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33493 every 8 connection(s)
I20260927 06:55:03.189015 7542 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:55:03.193938 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 7542
I20260927 06:55:03.194255 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 6879
Sep 27 06:55:03 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 27 06:55:03 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 27 06:55:03 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 06:55:03 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 06:55:03.226667 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.192939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53977 (local address 127.0.97.62:44789)
0927 06:55:03.202526 (+ 9587us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:03.202533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:03.204658 (+ 2125us) server_negotiation.cc:407] Connection header received
0927 06:55:03.205925 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:03.205933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:03.206045 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:03.206124 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:03.208457 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.209577 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.211947 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.212600 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.212690 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.217979 (+ 5289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:03.217987 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:03.217992 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:03.218081 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:03.222587 (+ 4506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.223749 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.223757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.223762 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.223955 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.224508 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.224513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.224517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.224857 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:03.224998 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:03.225982 (+ 984us) server_negotiation.cc:299] Negotiation successful
0927 06:55:03.226245 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9473,"thread_start_us":9380,"threads_started":1}
I20260927 06:55:03.229048 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.195530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.0.97.3:41593)
0927 06:55:03.198473 (+ 2943us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:03.198482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:03.205768 (+ 7286us) server_negotiation.cc:407] Connection header received
0927 06:55:03.206264 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:03.206274 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:03.206526 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:03.206641 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:03.208833 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.210289 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.212847 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.213658 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.213758 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.219678 (+ 5920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:03.219692 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:03.219710 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:03.219757 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:03.224230 (+ 4473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.225918 (+ 1688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.225926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.225930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.226108 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.226562 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.226567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.226570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.226962 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:03.227229 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:03.228375 (+ 1146us) server_negotiation.cc:299] Negotiation successful
0927 06:55:03.228644 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2770,"thread_start_us":2623,"threads_started":1}
I20260927 06:55:03.229331 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.192324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.2:53977)
0927 06:55:03.203046 (+ 10722us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:03.203103 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:03.205621 (+ 2518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:03.206592 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:03.206604 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:03.207371 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:03.208238 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.208264 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.209772 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.209777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:03.211772 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.211787 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.213011 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.213018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:03.213310 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.214461 (+ 1151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:03.214499 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:03.217639 (+ 3140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:03.222828 (+ 5189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:03.222838 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:03.222861 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:03.223577 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:03.224088 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:03.224095 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:03.224101 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:03.224370 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:03.225087 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:03.225098 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:03.225559 (+ 461us) client_negotiation.cc:769] Sending connection context
0927 06:55:03.227126 (+ 1567us) client_negotiation.cc:240] Negotiation successful
0927 06:55:03.227708 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10433,"thread_start_us":10256,"threads_started":1}
I20260927 06:55:03.230738 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.193284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33033 (local address 127.0.97.2:60641)
0927 06:55:03.205151 (+ 11867us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:03.205167 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:03.205987 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:03.213002 (+ 7015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:03.213006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:03.214107 (+ 1101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:03.214545 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.214561 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.216702 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.216708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:03.218292 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.218305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.219321 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.219326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:03.219524 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.220537 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:03.220565 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:03.223472 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:03.227401 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:03.227405 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:03.227408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:03.227987 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:03.228448 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:03.228452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:03.228456 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:03.228600 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:03.229275 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:03.229283 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:03.229427 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 06:55:03.229568 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 06:55:03.229817 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11743,"thread_start_us":11550,"threads_started":1}
I20260927 06:55:03.231047 7002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:03.229535 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.208163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38017 (local address 127.0.97.2:57801)
0927 06:55:03.212413 (+ 4250us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:03.212436 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:03.212581 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:03.213482 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:03.213486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:03.214040 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:03.214459 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.214476 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.216031 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.216036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:03.217628 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.217641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.218816 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.218823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:03.218994 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.219498 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:03.219518 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:03.222661 (+ 3143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:03.226656 (+ 3995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:03.226662 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:03.226666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:03.227259 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:03.227707 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:03.227711 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:03.227730 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:03.227891 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:03.228571 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:03.228576 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:03.228707 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 06:55:03.228844 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 06:55:03.229093 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4105,"spinlock_wait_cycles":13568,"thread_start_us":3958,"threads_started":1}
I20260927 06:55:03.231122 7671 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44789
I20260927 06:55:03.231436 7671 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:03.231555 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.205470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57801 (local address 127.0.97.60:38017)
0927 06:55:03.213005 (+ 7535us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:03.213011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:03.213028 (+ 17us) server_negotiation.cc:407] Connection header received
0927 06:55:03.213084 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:03.213090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:03.213166 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:03.213289 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:03.214682 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.215882 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.217796 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.218472 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.218579 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.222936 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:03.222944 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:03.222950 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:03.222993 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:03.226517 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.227425 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.227430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.227433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.227568 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.228024 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.228029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.228031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.228362 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:03.228527 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:03.229729 (+ 1202us) server_negotiation.cc:299] Negotiation successful
0927 06:55:03.229977 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7416,"thread_start_us":7337,"threads_started":1}
I20260927 06:55:03.231976 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.191788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60641 (local address 127.0.97.61:33033)
0927 06:55:03.212106 (+ 20318us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:03.212112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:03.212129 (+ 17us) server_negotiation.cc:407] Connection header received
0927 06:55:03.212181 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:03.212186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:03.212271 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:03.213638 (+ 1367us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:03.214762 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.216549 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.218474 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.219097 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.219236 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.223687 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:03.223694 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:03.223700 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:03.223742 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:03.227241 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.228128 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.228135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.228139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.228309 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.228728 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.228734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.228737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.229077 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:03.229222 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:03.231089 (+ 1867us) server_negotiation.cc:299] Negotiation successful
0927 06:55:03.231449 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20206,"thread_start_us":20082,"threads_started":1}
I20260927 06:55:03.232306 7671 heartbeater.cc:511] Master 127.0.97.62:44789 requested a full tablet report, sending...
I20260927 06:55:03.232692 7670 heartbeater.cc:347] Connected to a master server at 127.0.97.60:38017
I20260927 06:55:03.232818 7670 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:03.233243 7670 heartbeater.cc:511] Master 127.0.97.60:38017 requested a full tablet report, sending...
I20260927 06:55:03.233366 7672 heartbeater.cc:347] Connected to a master server at 127.0.97.61:33033
I20260927 06:55:03.233528 7672 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:03.234036 7672 heartbeater.cc:511] Master 127.0.97.61:33033 requested a full tablet report, sending...
I20260927 06:55:03.234341 6551 ts_manager.cc:194] Re-registered known tserver with Master: 151233c285ec4937b55164d138143479 (127.0.97.2:33493)
I20260927 06:55:03.235119 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53977
I20260927 06:55:03.236577 7350 ts_manager.cc:194] Registered new tserver with Master: 151233c285ec4937b55164d138143479 (127.0.97.2:33493)
I20260927 06:55:03.237444 7155 ts_manager.cc:194] Registered new tserver with Master: 151233c285ec4937b55164d138143479 (127.0.97.2:33493)
I20260927 06:55:03.316496 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 6879
I20260927 06:55:03.333401 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41593
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44789
--webserver_port=33639
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:44789,127.0.97.61:33033
--tserver_master_addrs=127.0.97.62:44789,127.0.97.61:33033,127.0.97.60:38017 with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:55:03.645665 7683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:03.645951 7683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:03.646121 7683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:03.653424 7683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:03.653558 7683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:03.653771 7683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:55:03.663843 7683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33639
--tserver_master_addrs=127.0.97.62:44789,127.0.97.61:33033,127.0.97.60:38017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:03.665056 7683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:03.665210 7531 heartbeater.cc:503] Master 127.0.97.62:44789 was elected leader, sending a full tablet report...
I20260927 06:55:03.666493 7683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:03.679066 7689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:03.679116 7691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:03.679138 7688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:03.680085 7683 server_base.cc:1034] running on GCE node
I20260927 06:55:03.680639 7683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:03.681463 7683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:03.682665 7683 hybrid_clock.cc:648] HybridClock initialized: now 1790492103682631 us; error 41 us; skew 500 ppm
I20260927 06:55:03.682952 7683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:03 dist-test-slave-f892 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:03.685853 7683 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 06:55:03.687167 7683 webserver.cc:460] Webserver started at http://127.0.97.3:33639/ using document root <none> and password file <none>
I20260927 06:55:03.687768 7683 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:03.687875 7683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:03.692764 7683 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 06:55:03.695159 7698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:03.696322 7683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 06:55:03.696472 7683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "9eebd909c11f4326bb832093502f823f"
format_stamp: "Formatted at 2026-09-27 06:55:00 on dist-test-slave-f892"
server_key: "b4a7b9aa97caf888bf51c5f6286dbfac"
server_key_iv: "92c68ce4239fc0f78bb37163715ab994"
server_key_version: "encryptionkey@0"
I20260927 06:55:03.697324 7683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:03.721632 7683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:03.725723 7683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:03.726068 7683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:03.727005 7683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:03.728359 7683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:03.728439 7683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:03.728587 7683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:03.728678 7683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:03.761471 7683 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41593
I20260927 06:55:03.761570 7810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41593 every 8 connection(s)
I20260927 06:55:03.762892 7683 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:55:03.771816 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 7683
Sep 27 06:55:03 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 06:55:03 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 06:55:03.798036 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.766193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38917 (local address 127.0.97.61:33033)
0927 06:55:03.766637 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:03.766645 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:03.770564 (+ 3919us) server_negotiation.cc:407] Connection header received
0927 06:55:03.770804 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:03.770812 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:03.770922 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:03.771033 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:03.774378 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.777181 (+ 2803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.780707 (+ 3526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.781566 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.781700 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.789073 (+ 7373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:03.789080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:03.789085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:03.789130 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:03.793784 (+ 4654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.794835 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.794843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.794849 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.795024 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.795575 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.795583 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.795588 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.795991 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:03.796160 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:03.797211 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0927 06:55:03.797465 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":473,"threads_started":1}
Sep 27 06:55:03 dist-test-slave-f892 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 06:55:03.800604 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.766409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33033 (local address 127.0.97.3:38917)
0927 06:55:03.770480 (+ 4071us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:03.770499 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:03.770644 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:03.773049 (+ 2405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:03.773057 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:03.773773 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:03.774177 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.774192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.777456 (+ 3264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.777465 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:03.780510 (+ 3045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.780529 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.783360 (+ 2831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.783373 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:03.783631 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.784600 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:03.784917 (+ 317us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:03.788839 (+ 3922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:03.793962 (+ 5123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:03.793974 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:03.793994 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:03.794689 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:03.795174 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:03.795180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:03.795186 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:03.795435 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:03.796253 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:03.796262 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:03.796704 (+ 442us) client_negotiation.cc:769] Sending connection context
0927 06:55:03.798343 (+ 1639us) client_negotiation.cc:240] Negotiation successful
0927 06:55:03.798931 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3940,"mutex_wait_us":186,"thread_start_us":3758,"threads_started":1}
I20260927 06:55:03.802294 7813 heartbeater.cc:347] Connected to a master server at 127.0.97.61:33033
I20260927 06:55:03.802619 7813 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:03.803467 7813 heartbeater.cc:511] Master 127.0.97.61:33033 requested a full tablet report, sending...
I20260927 06:55:03.805397 7155 ts_manager.cc:194] Registered new tserver with Master: 9eebd909c11f4326bb832093502f823f (127.0.97.3:41593)
I20260927 06:55:03.812546 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.765622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58913 (local address 127.0.97.62:44789)
0927 06:55:03.766120 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:03.766126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:03.767257 (+ 1131us) server_negotiation.cc:407] Connection header received
0927 06:55:03.769248 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:03.769257 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:03.769362 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:03.769508 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:03.771258 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.772774 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.781967 (+ 9193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.782861 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.783019 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.806783 (+ 23764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:03.806790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:03.806794 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:03.806831 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:03.809507 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.810365 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.810371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.810384 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.810545 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.810993 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.811000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.811005 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.811325 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:03.811477 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:03.811944 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 06:55:03.812128 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":244,"threads_started":1}
I20260927 06:55:03.812567 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.766277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.3:58913)
0927 06:55:03.767100 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:03.767143 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:03.769008 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:03.769669 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:03.769677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:03.770262 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:03.771023 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.771054 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.777456 (+ 6402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.777465 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:03.781082 (+ 3617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.781101 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.783196 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.783206 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:03.783593 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.785070 (+ 1477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:03.785099 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:03.806592 (+ 21493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:03.809659 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:03.809664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:03.809668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:03.810248 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:03.810667 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:03.810674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:03.810680 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:03.810868 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:03.811528 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:03.811532 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:03.811688 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 06:55:03.811853 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 06:55:03.812123 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":1762,"thread_start_us":386,"threads_started":1}
I20260927 06:55:03.813845 7812 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44789
I20260927 06:55:03.813968 7812 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:03.814407 7812 heartbeater.cc:511] Master 127.0.97.62:44789 requested a full tablet report, sending...
I20260927 06:55:03.814893 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.770747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44219 (local address 127.0.97.60:38017)
0927 06:55:03.786270 (+ 15523us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:03.786277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:03.787220 (+ 943us) server_negotiation.cc:407] Connection header received
0927 06:55:03.787370 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:03.787378 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:03.788795 (+ 1417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:03.788902 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:03.790474 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.791903 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.794581 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.795275 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.795384 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.799904 (+ 4520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:03.799913 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:03.799919 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:03.799964 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:03.803747 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.811032 (+ 7285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.811039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.811041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.811155 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.812829 (+ 1674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.812833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.812836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.813395 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:03.813556 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:03.814109 (+ 553us) server_negotiation.cc:299] Negotiation successful
0927 06:55:03.814389 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15386,"thread_start_us":15523,"threads_started":1}
I20260927 06:55:03.815356 6551 ts_manager.cc:194] Re-registered known tserver with Master: 9eebd909c11f4326bb832093502f823f (127.0.97.3:41593)
I20260927 06:55:03.815920 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.770602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38017 (local address 127.0.97.3:44219)
0927 06:55:03.787104 (+ 16502us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:03.787126 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:03.787242 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:03.789107 (+ 1865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:03.789115 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:03.789825 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:03.790258 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.790273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.792117 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.792125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:03.794395 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.794414 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.795731 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.795737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:03.795965 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.796551 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:03.796579 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:03.799661 (+ 3082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:03.809663 (+ 10002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:03.809671 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:03.809677 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:03.810563 (+ 886us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:03.811285 (+ 722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:03.811289 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:03.811293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:03.811437 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:03.813643 (+ 2206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:03.813647 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:03.813792 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 06:55:03.815086 (+ 1294us) client_negotiation.cc:240] Negotiation successful
0927 06:55:03.815438 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16377,"spinlock_wait_cycles":27904,"thread_start_us":16226,"threads_started":1}
I20260927 06:55:03.816201 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58913
I20260927 06:55:03.816684 7811 heartbeater.cc:347] Connected to a master server at 127.0.97.60:38017
I20260927 06:55:03.816825 7811 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:03.817672 7811 heartbeater.cc:511] Master 127.0.97.60:38017 requested a full tablet report, sending...
I20260927 06:55:03.818626 7350 ts_manager.cc:194] Registered new tserver with Master: 9eebd909c11f4326bb832093502f823f (127.0.97.3:41593)
I20260927 06:55:03.831694 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.789350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.0.97.62:44789)
0927 06:55:03.807286 (+ 17936us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:03.807294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:03.807317 (+ 23us) server_negotiation.cc:407] Connection header received
0927 06:55:03.807381 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:03.807388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:03.807481 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:03.807605 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:03.810426 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.812482 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.819202 (+ 6720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.819601 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.819692 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.823353 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:03.823359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:03.823365 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:03.823414 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:03.827335 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.828169 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.828174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.828177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.828313 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.829951 (+ 1638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.829954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.829956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.830349 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:03.830588 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:03.830908 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 06:55:03.831121 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17763,"thread_start_us":17347,"threads_started":1}
I20260927 06:55:03.832141 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.802121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.0.97.60:38017)
0927 06:55:03.813771 (+ 11650us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:03.813777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:03.813791 (+ 14us) server_negotiation.cc:407] Connection header received
0927 06:55:03.813834 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:03.813839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:03.813915 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:03.814007 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:03.819260 (+ 5253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.820310 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.821683 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.822179 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.822296 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.825543 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:03.825550 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:03.825554 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:03.825594 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:03.828975 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.829612 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.829618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.829622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.829779 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.830240 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.830244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.830246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.830536 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:03.830743 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:03.831594 (+ 851us) server_negotiation.cc:299] Negotiation successful
0927 06:55:03.831784 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11525,"thread_start_us":11405,"threads_started":1}
I20260927 06:55:03.835211 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.789036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.0.97.61:33033)
0927 06:55:03.798273 (+ 9237us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:03.798277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:03.805456 (+ 7179us) server_negotiation.cc:407] Connection header received
0927 06:55:03.811687 (+ 6231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:03.811694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:03.811837 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:03.811953 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:03.822798 (+ 10845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.823681 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.825138 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.825665 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.825803 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.828496 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:03.828503 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:03.828507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:03.828543 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:03.831584 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.832625 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.832631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.832633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.833335 (+ 702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:03.833767 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:03.833773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:03.833776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:03.834172 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:03.834397 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:03.834689 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 06:55:03.834897 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9137,"thread_start_us":26821,"threads_started":1}
I20260927 06:55:03.835799 6551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:55:03.843158 6551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:55:03.882634 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.860886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33462 (local address 127.0.97.2:33493)
0927 06:55:03.862406 (+ 1520us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:03.862437 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:03.874625 (+ 12188us) server_negotiation.cc:407] Connection header received
0927 06:55:03.875193 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:03.875210 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:03.875516 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:03.875744 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:03.876316 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.877711 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.880675 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.881548 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.881692 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.881911 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:03.882139 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 06:55:03.882336 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1348,"thread_start_us":1249,"threads_started":1}
I20260927 06:55:03.884630 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.860672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33493 (local address 127.0.0.1:33462)
0927 06:55:03.874513 (+ 13841us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:03.874529 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:03.874641 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:03.875724 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:03.875727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:03.875747 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:03.876128 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.876146 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.878073 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.878078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:03.880513 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.880524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.881730 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.881735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:03.881917 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.881959 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 06:55:03.882101 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 06:55:03.884204 (+ 2103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13723,"mutex_wait_us":89,"thread_start_us":13569,"threads_started":1}
I20260927 06:55:03.884747 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.860188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36787 (local address 127.0.0.1:46128)
0927 06:55:03.872737 (+ 12549us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:03.872769 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:03.872921 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:03.876004 (+ 3083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:03.876012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:03.876052 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:03.876534 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.876549 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.878194 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.878202 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:03.882046 (+ 3844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.882065 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.883977 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.883985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:03.884166 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.884199 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 06:55:03.884294 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 06:55:03.884405 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6089,"mutex_wait_us":32,"thread_start_us":5935,"threads_started":1}
I20260927 06:55:03.884817 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.860494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46128 (local address 127.0.97.1:36787)
0927 06:55:03.873495 (+ 13001us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:03.873508 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:03.873556 (+ 48us) server_negotiation.cc:407] Connection header received
0927 06:55:03.873680 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:03.873691 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:03.874951 (+ 1260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:03.875180 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:03.876734 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.878077 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.882198 (+ 4121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.883748 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.883876 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.884077 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:03.884347 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 06:55:03.884502 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12769,"thread_start_us":14226,"threads_started":1}
I20260927 06:55:03.888329 7605 tablet_service.cc:1467] Processing CreateTablet for tablet 8d7103e63dbe46ef841a67e8832861be (DEFAULT_TABLE table=test-table [id=442ce06c2f734700b0bca5b73d5032e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:03.888705 7465 tablet_service.cc:1467] Processing CreateTablet for tablet 8d7103e63dbe46ef841a67e8832861be (DEFAULT_TABLE table=test-table [id=442ce06c2f734700b0bca5b73d5032e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:03.889780 7605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d7103e63dbe46ef841a67e8832861be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:03.889931 7465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d7103e63dbe46ef841a67e8832861be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:03.891211 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.866473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41593 (local address 127.0.0.1:44820)
0927 06:55:03.883153 (+ 16680us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:03.883166 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:03.883295 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:03.885010 (+ 1715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:03.885015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:03.885036 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:03.885269 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.885276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.886904 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.886912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:03.888273 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.888288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.890459 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.890465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:03.890636 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.890667 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 06:55:03.890763 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 06:55:03.890882 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16560,"thread_start_us":16449,"threads_started":1}
I20260927 06:55:03.891259 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.866753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44820 (local address 127.0.97.3:41593)
0927 06:55:03.882778 (+ 16025us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:03.882790 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:03.883260 (+ 470us) server_negotiation.cc:407] Connection header received
0927 06:55:03.883563 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:03.883569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:03.883728 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:03.883919 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:03.885443 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.886774 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.888402 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.890265 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.890404 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:03.890598 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:03.890809 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 06:55:03.890961 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15821,"thread_start_us":16033,"threads_started":1}
I20260927 06:55:03.894336 7746 tablet_service.cc:1467] Processing CreateTablet for tablet 8d7103e63dbe46ef841a67e8832861be (DEFAULT_TABLE table=test-table [id=442ce06c2f734700b0bca5b73d5032e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:03.895411 7746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d7103e63dbe46ef841a67e8832861be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:03.904790 7849 tablet_bootstrap.cc:492] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40: Bootstrap starting.
I20260927 06:55:03.906107 7848 tablet_bootstrap.cc:492] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479: Bootstrap starting.
I20260927 06:55:03.907407 7850 tablet_bootstrap.cc:492] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f: Bootstrap starting.
I20260927 06:55:03.909001 7848 tablet_bootstrap.cc:654] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:03.909829 7850 tablet_bootstrap.cc:654] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:03.909916 7848 log.cc:824] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479: Log is configured to *not* fsync() on all Append() calls
I20260927 06:55:03.911015 7850 log.cc:824] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f: Log is configured to *not* fsync() on all Append() calls
I20260927 06:55:03.912837 7848 tablet_bootstrap.cc:492] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479: No bootstrap required, opened a new log
I20260927 06:55:03.913201 7848 ts_tablet_manager.cc:1397] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 06:55:03.913527 7849 tablet_bootstrap.cc:654] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:03.914278 7850 tablet_bootstrap.cc:492] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f: No bootstrap required, opened a new log
I20260927 06:55:03.914693 7849 log.cc:824] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40: Log is configured to *not* fsync() on all Append() calls
I20260927 06:55:03.914754 7850 ts_tablet_manager.cc:1397] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 06:55:03.920734 7848 raft_consensus.cc:348] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } } peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } }
I20260927 06:55:03.921098 7848 raft_consensus.cc:374] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:03.921214 7848 raft_consensus.cc:729] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 151233c285ec4937b55164d138143479, State: Initialized, Role: FOLLOWER
I20260927 06:55:03.922338 7848 consensus_queue.cc:260] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } } peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } }
I20260927 06:55:03.924034 7848 ts_tablet_manager.cc:1428] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20260927 06:55:03.924285 7671 heartbeater.cc:503] Master 127.0.97.62:44789 was elected leader, sending a full tablet report...
I20260927 06:55:03.924487 7849 tablet_bootstrap.cc:492] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40: No bootstrap required, opened a new log
I20260927 06:55:03.924747 7849 ts_tablet_manager.cc:1397] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.000s
I20260927 06:55:03.926937 7850 raft_consensus.cc:348] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } } peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } }
I20260927 06:55:03.927350 7850 raft_consensus.cc:374] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:03.927469 7850 raft_consensus.cc:729] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eebd909c11f4326bb832093502f823f, State: Initialized, Role: FOLLOWER
I20260927 06:55:03.928064 7850 consensus_queue.cc:260] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } } peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } }
I20260927 06:55:03.930104 7812 heartbeater.cc:503] Master 127.0.97.62:44789 was elected leader, sending a full tablet report...
I20260927 06:55:03.930717 7850 ts_tablet_manager.cc:1428] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260927 06:55:03.937127 7849 raft_consensus.cc:348] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } } peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } }
I20260927 06:55:03.937453 7849 raft_consensus.cc:374] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:03.937546 7849 raft_consensus.cc:729] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3b556cc4054b7ca08c3abe1a70ae40, State: Initialized, Role: FOLLOWER
I20260927 06:55:03.938131 7849 consensus_queue.cc:260] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } } peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } }
I20260927 06:55:03.939805 7849 ts_tablet_manager.cc:1428] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
W20260927 06:55:03.947474 7673 tablet.cc:2369] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:55:03.992230 7856 raft_consensus.cc:482] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:55:03.992612 7856 raft_consensus.cc:504] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } } peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } }
I20260927 06:55:03.994302 7856 leader_election.cc:288] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 151233c285ec4937b55164d138143479 (127.0.97.2:33493), 9eebd909c11f4326bb832093502f823f (127.0.97.3:41593)
I20260927 06:55:04.002684 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.994791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38285 (local address 127.0.97.3:41593)
0927 06:55:03.995037 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:03.995047 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:03.995883 (+ 836us) server_negotiation.cc:407] Connection header received
0927 06:55:03.996051 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:03.996060 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:03.996146 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:03.996258 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:03.996788 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.997806 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:04.000977 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:04.001735 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:04.001872 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:04.001976 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:04.002318 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 06:55:04.002448 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 06:55:04.002815 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.995035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55461 (local address 127.0.97.2:33493)
0927 06:55:03.995284 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:03.995293 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:03.996457 (+ 1164us) server_negotiation.cc:407] Connection header received
0927 06:55:03.996639 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:03.996643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:03.996711 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:03.996790 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:03.997378 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.998208 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:04.000930 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:04.001863 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:04.002009 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:04.002128 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:04.002452 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 06:55:04.002572 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 06:55:04.003150 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.994698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41593 (local address 127.0.97.1:38285)
0927 06:55:03.995790 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:03.995806 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:03.995919 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:03.996302 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:03.996306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:03.996334 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:03.996664 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.996672 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.997939 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.997949 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:04.000779 (+ 2830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:04.000792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:04.001942 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:04.001948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:04.002162 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:04.002185 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 06:55:04.002297 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 06:55:04.002428 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":109,"spinlock_wait_cycles":1408,"thread_start_us":1397,"threads_started":1}
I20260927 06:55:04.003299 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:03.994708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33493 (local address 127.0.97.1:55461)
0927 06:55:03.996351 (+ 1643us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:03.996368 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:03.996494 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:03.996823 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:03.996828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:03.996854 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:03.997186 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:03.997196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:03.998647 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:03.998652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:04.000778 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:04.000792 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:04.002089 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:04.002096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:04.002290 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:04.002312 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 06:55:04.002429 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 06:55:04.002552 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1488,"mutex_wait_us":33,"thread_start_us":796,"threads_started":1}
I20260927 06:55:04.004046 7766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d7103e63dbe46ef841a67e8832861be" candidate_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eebd909c11f4326bb832093502f823f" is_pre_election: true
I20260927 06:55:04.004073 7625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d7103e63dbe46ef841a67e8832861be" candidate_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "151233c285ec4937b55164d138143479" is_pre_election: true
I20260927 06:55:04.004482 7766 raft_consensus.cc:2454] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b3b556cc4054b7ca08c3abe1a70ae40 in term 0.
I20260927 06:55:04.004508 7625 raft_consensus.cc:2454] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b3b556cc4054b7ca08c3abe1a70ae40 in term 0.
I20260927 06:55:04.005065 7420 leader_election.cc:302] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b3b556cc4054b7ca08c3abe1a70ae40, 9eebd909c11f4326bb832093502f823f; no voters:
I20260927 06:55:04.005419 7856 raft_consensus.cc:2781] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:55:04.005561 7856 raft_consensus.cc:482] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:55:04.005659 7856 raft_consensus.cc:3037] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:04.007743 7856 raft_consensus.cc:504] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } } peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } }
I20260927 06:55:04.008248 7856 leader_election.cc:288] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [CANDIDATE]: Term 1 election: Requested vote from peers 151233c285ec4937b55164d138143479 (127.0.97.2:33493), 9eebd909c11f4326bb832093502f823f (127.0.97.3:41593)
I20260927 06:55:04.008666 7766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d7103e63dbe46ef841a67e8832861be" candidate_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eebd909c11f4326bb832093502f823f"
I20260927 06:55:04.008652 7625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d7103e63dbe46ef841a67e8832861be" candidate_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "151233c285ec4937b55164d138143479"
I20260927 06:55:04.008842 7766 raft_consensus.cc:3037] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:04.008857 7625 raft_consensus.cc:3037] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:04.011189 7766 raft_consensus.cc:2454] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b3b556cc4054b7ca08c3abe1a70ae40 in term 1.
I20260927 06:55:04.011439 7625 raft_consensus.cc:2454] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b3b556cc4054b7ca08c3abe1a70ae40 in term 1.
I20260927 06:55:04.011543 7420 leader_election.cc:302] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b3b556cc4054b7ca08c3abe1a70ae40, 9eebd909c11f4326bb832093502f823f; no voters:
I20260927 06:55:04.011819 7856 raft_consensus.cc:2781] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:55:04.012207 7856 raft_consensus.cc:686] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 1 LEADER]: Becoming Leader. State: Replica: 8b3b556cc4054b7ca08c3abe1a70ae40, State: Running, Role: LEADER
I20260927 06:55:04.012722 7856 consensus_queue.cc:237] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } } peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } }
W20260927 06:55:04.016115 7814 tablet.cc:2369] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:55:04.017843 6551 catalog_manager.cc:5495] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b3b556cc4054b7ca08c3abe1a70ae40 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } health_report { overall_health: UNKNOWN } } }
I20260927 06:55:04.038230 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:04.032843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.0.97.1:36787)
0927 06:55:04.033059 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:04.033064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:04.033083 (+ 19us) server_negotiation.cc:407] Connection header received
0927 06:55:04.033270 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:04.033273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:04.033351 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:04.033467 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:55:04.033754 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:04.034497 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:04.035778 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:04.036628 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:04.036771 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:04.037145 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:55:04.037636 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:55:04.037730 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:04.037854 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 06:55:04.037930 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 06:55:04.045648 7625 raft_consensus.cc:1261] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479 [term 1 FOLLOWER]: Refusing update from remote peer 8b3b556cc4054b7ca08c3abe1a70ae40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:55:04.045656 7766 raft_consensus.cc:1261] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f [term 1 FOLLOWER]: Refusing update from remote peer 8b3b556cc4054b7ca08c3abe1a70ae40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:55:04.046630 7859 consensus_queue.cc:1035] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:55:04.046931 7856 consensus_queue.cc:1035] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:55:04.059175 7863 mvcc.cc:205] Tried to move back new op lower bound from 7333855658161934336 to 7333855658039656448. Current Snapshot: MvccSnapshot[applied={T|T < 7333855658161934336}]
I20260927 06:55:04.060824 7862 mvcc.cc:205] Tried to move back new op lower bound from 7333855658161934336 to 7333855658039656448. Current Snapshot: MvccSnapshot[applied={T|T < 7333855658161934336 or (T in {7333855658161934336})}]
I20260927 06:55:04.071269 7864 mvcc.cc:205] Tried to move back new op lower bound from 7333855658161934336 to 7333855658039656448. Current Snapshot: MvccSnapshot[applied={T|T < 7333855658161934336}]
I20260927 06:55:04.072487 6551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:55:04.072714 6551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:55:04.080930 7605 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d7103e63dbe46ef841a67e8832861be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33462
I20260927 06:55:04.084583 6551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f346c97981314f6c97c333fb51cf4e55: Sending DeleteTablet for 3 replicas of tablet 8d7103e63dbe46ef841a67e8832861be
I20260927 06:55:04.085413 7746 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d7103e63dbe46ef841a67e8832861be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44820
I20260927 06:55:04.086359 7465 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d7103e63dbe46ef841a67e8832861be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46128
I20260927 06:55:04.086323 7604 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d7103e63dbe46ef841a67e8832861be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33462
I20260927 06:55:04.087566 7873 tablet_replica.cc:330] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40: stopping tablet replica
I20260927 06:55:04.087635 7745 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d7103e63dbe46ef841a67e8832861be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44820
I20260927 06:55:04.088177 7873 raft_consensus.cc:2229] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:55:04.088938 7873 raft_consensus.cc:2258] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:04.094602 7871 tablet_replica.cc:330] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479: stopping tablet replica
I20260927 06:55:04.094806 7873 ts_tablet_manager.cc:1905] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:04.094982 7871 raft_consensus.cc:2229] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:55:04.095759 7871 raft_consensus.cc:2258] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:04.096495 7872 tablet_replica.cc:330] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f: stopping tablet replica
I20260927 06:55:04.096807 7872 raft_consensus.cc:2229] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:55:04.097216 7872 raft_consensus.cc:2258] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:04.098675 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 6519
I20260927 06:55:04.099067 7871 ts_tablet_manager.cc:1905] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:04.099712 6581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:04.105787 7871 ts_tablet_manager.cc:1918] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:55:04.105953 7871 log.cc:1196] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/8d7103e63dbe46ef841a67e8832861be
I20260927 06:55:04.106503 7871 ts_tablet_manager.cc:1939] T 8d7103e63dbe46ef841a67e8832861be P 151233c285ec4937b55164d138143479: Deleting consensus metadata
I20260927 06:55:04.110587 7872 ts_tablet_manager.cc:1905] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:04.112823 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:04.090036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.0.97.61:33033)
0927 06:55:04.090193 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:04.090198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:04.090218 (+ 20us) server_negotiation.cc:407] Connection header received
0927 06:55:04.090258 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:04.090262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:04.090313 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:04.090415 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:04.092573 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:04.093394 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:04.096031 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:04.096602 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:04.096740 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:04.104403 (+ 7663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:04.104413 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:04.104421 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:04.104476 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:04.109012 (+ 4536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:04.109824 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:04.109832 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:04.109838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:04.110043 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:04.110505 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:04.110513 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:04.110519 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:04.111407 (+ 888us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:04.111730 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:04.112023 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 06:55:04.112320 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 06:55:04.114807 7873 ts_tablet_manager.cc:1918] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:55:04.115021 7873 log.cc:1196] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/8d7103e63dbe46ef841a67e8832861be
I20260927 06:55:04.115716 7873 ts_tablet_manager.cc:1939] T 8d7103e63dbe46ef841a67e8832861be P 8b3b556cc4054b7ca08c3abe1a70ae40: Deleting consensus metadata
I20260927 06:55:04.123575 7872 ts_tablet_manager.cc:1918] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:55:04.123764 7872 log.cc:1196] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/8d7103e63dbe46ef841a67e8832861be
I20260927 06:55:04.124390 7872 ts_tablet_manager.cc:1939] T 8d7103e63dbe46ef841a67e8832861be P 9eebd909c11f4326bb832093502f823f: Deleting consensus metadata
I20260927 06:55:04.129161 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:04.091240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.0.97.60:38017)
0927 06:55:04.091409 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:04.091415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:04.112670 (+ 21255us) server_negotiation.cc:407] Connection header received
0927 06:55:04.112925 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:04.112933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:04.113020 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:04.113089 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:04.114868 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:04.115974 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:04.117866 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:04.118547 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:04.118656 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:04.122986 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:04.122993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:04.122998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:04.123039 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:04.126560 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:04.127238 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:04.127243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:04.127247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:04.127375 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:04.127752 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:04.127757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:04.127760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:04.128047 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:04.128288 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:04.128602 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 06:55:04.128824 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 06:55:04.267535 6536 catalog_manager.cc:4808] TS 151233c285ec4937b55164d138143479 (127.0.97.2:33493): delete failed for tablet 8d7103e63dbe46ef841a67e8832861be because the tablet was not found. No further retry: Not found: Tablet not found: 8d7103e63dbe46ef841a67e8832861be
I20260927 06:55:04.268416 6537 catalog_manager.cc:4841] TS 8b3b556cc4054b7ca08c3abe1a70ae40 (127.0.97.1:36787): tablet 8d7103e63dbe46ef841a67e8832861be (table test-table [id=442ce06c2f734700b0bca5b73d5032e9]) successfully deleted
W20260927 06:55:04.268617 6537 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.1:36787, user_credentials={real_user=slave}} blocked reactor thread for 149747us
I20260927 06:55:04.268962 6537 catalog_manager.cc:4841] TS 9eebd909c11f4326bb832093502f823f (127.0.97.3:41593): tablet 8d7103e63dbe46ef841a67e8832861be (table test-table [id=442ce06c2f734700b0bca5b73d5032e9]) successfully deleted
W20260927 06:55:04.269193 6537 catalog_manager.cc:4808] TS 9eebd909c11f4326bb832093502f823f (127.0.97.3:41593): delete failed for tablet 8d7103e63dbe46ef841a67e8832861be because the tablet was not found. No further retry: Not found: Tablet not found: 8d7103e63dbe46ef841a67e8832861be
I20260927 06:55:04.270241 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 6519
I20260927 06:55:04.319061 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:04.303522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.0.97.60:38017)
0927 06:55:04.303694 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:04.303703 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:04.303896 (+ 193us) server_negotiation.cc:407] Connection header received
0927 06:55:04.304073 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:04.304081 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:04.304179 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:04.304254 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:04.306552 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:04.307269 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:04.309134 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:04.309623 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:04.309753 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:04.312393 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:04.312399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:04.312402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:04.312433 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:04.315937 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:04.316855 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:04.316860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:04.316864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:04.316991 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:04.317350 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:04.317356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:04.317359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:04.317745 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:04.317963 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:04.318522 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 06:55:04.318735 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 06:55:04.319855 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:04.302220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.0.97.61:33033)
0927 06:55:04.302453 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:04.302463 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:04.303508 (+ 1045us) server_negotiation.cc:407] Connection header received
0927 06:55:04.303780 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:04.303787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:04.303865 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:04.303960 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:04.306130 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:04.307255 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:04.308556 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:04.309024 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:04.309159 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:04.312747 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:04.312752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:04.312757 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:04.312798 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:04.315964 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:04.316847 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:04.316853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:04.316857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:04.316993 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:04.317844 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:04.317850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:04.317853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:04.318219 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:04.318455 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:04.319030 (+ 575us) server_negotiation.cc:299] Negotiation successful
0927 06:55:04.319519 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 06:55:05.072042 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.071315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.1:36077)
0927 06:55:05.071777 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.071889 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":228,"threads_started":1}
W20260927 06:55:05.072474 7531 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44789 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
I20260927 06:55:05.082983 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.082403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.2:57341)
0927 06:55:05.082757 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.082847 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":191,"threads_started":1}
W20260927 06:55:05.083483 7671 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44789 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
I20260927 06:55:05.087023 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.086418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.3:39603)
0927 06:55:05.086771 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.086869 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":186,"threads_started":1}
W20260927 06:55:05.087353 7812 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44789 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
I20260927 06:55:05.580821 7885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f346c97981314f6c97c333fb51cf4e55)
I20260927 06:55:05.581054 7885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: false } }
I20260927 06:55:05.582058 7886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f346c97981314f6c97c333fb51cf4e55)
I20260927 06:55:05.582311 7886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: false } }
I20260927 06:55:05.582625 7885 leader_election.cc:288] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f346c97981314f6c97c333fb51cf4e55 (127.0.97.62:44789), 9d32914ce3024b72a55b8703f765deba (127.0.97.61:33033)
I20260927 06:55:05.593819 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.582709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.0.1:34568)
0927 06:55:05.593408 (+ 10699us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.593567 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10579,"thread_start_us":10448,"threads_started":1}
W20260927 06:55:05.594096 7333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
I20260927 06:55:05.595860 7886 leader_election.cc:288] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f346c97981314f6c97c333fb51cf4e55 (127.0.97.62:44789), 30e8b6e93caf4f649f2c446ede4d7957 (127.0.97.60:38017)
I20260927 06:55:05.597041 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.583774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.0.1:34572)
0927 06:55:05.596492 (+ 12718us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.596674 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12599,"threads_started":1}
I20260927 06:55:05.603710 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.582810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33033 (local address 127.0.0.1:32886)
0927 06:55:05.594034 (+ 11224us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.594051 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:05.594185 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:05.594864 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:05.594873 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:05.594912 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:05.595191 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:05.595204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.597186 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.597195 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:05.600009 (+ 2814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:05.600027 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.602651 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.602660 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:05.602917 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:05.602970 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 06:55:05.603077 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 06:55:05.603276 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11122,"thread_start_us":12783,"threads_started":1}
W20260927 06:55:05.604508 7139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
I20260927 06:55:05.604897 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.582967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.0.97.61:33033)
0927 06:55:05.593653 (+ 10686us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:05.593659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:05.594300 (+ 641us) server_negotiation.cc:407] Connection header received
0927 06:55:05.594395 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:05.594400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:05.594512 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:05.594638 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:05.595410 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.596963 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.600228 (+ 3265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.602173 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.602340 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:05.602524 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:05.604019 (+ 1495us) server_negotiation.cc:299] Negotiation successful
0927 06:55:05.604231 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":10582,"thread_start_us":10663,"threads_started":1}
I20260927 06:55:05.605463 7165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30e8b6e93caf4f649f2c446ede4d7957" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d32914ce3024b72a55b8703f765deba" is_pre_election: true
I20260927 06:55:05.605789 7165 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e8b6e93caf4f649f2c446ede4d7957 in term 1.
I20260927 06:55:05.606323 7336 leader_election.cc:302] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e8b6e93caf4f649f2c446ede4d7957, 9d32914ce3024b72a55b8703f765deba; no voters:
I20260927 06:55:05.606741 7885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 06:55:05.606863 7885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f346c97981314f6c97c333fb51cf4e55)
I20260927 06:55:05.606997 7885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 1 FOLLOWER]: Advancing to term 2
I20260927 06:55:05.607098 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.590168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38017 (local address 127.0.0.1:38178)
0927 06:55:05.597284 (+ 7116us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.597300 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:05.597412 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:05.598165 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:05.598173 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:05.598206 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:05.598593 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:05.598607 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.600618 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.600627 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:05.603608 (+ 2981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:05.603627 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.606296 (+ 2669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.606303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:05.606483 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:05.606522 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 06:55:05.606612 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 06:55:05.606744 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7033,"mutex_wait_us":1,"thread_start_us":6083,"threads_started":1}
I20260927 06:55:05.607450 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.607099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.0.1:34608)
0927 06:55:05.607231 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.607308 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 06:55:05.607615 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.606745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.0.1:34592)
0927 06:55:05.607134 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.607424 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260927 06:55:05.607720 7333 leader_election.cc:334] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f346c97981314f6c97c333fb51cf4e55 (127.0.97.62:44789): Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
W20260927 06:55:05.607848 7139 leader_election.cc:334] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f346c97981314f6c97c333fb51cf4e55 (127.0.97.62:44789): Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
I20260927 06:55:05.608083 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.592143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.0.97.60:38017)
0927 06:55:05.593191 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:05.593204 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:05.597540 (+ 4336us) server_negotiation.cc:407] Connection header received
0927 06:55:05.597621 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:05.597628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:05.597717 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:05.597820 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:05.598828 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.600436 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.604363 (+ 3927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.606121 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.606237 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:05.606368 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:05.606690 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 06:55:05.606810 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":743,"threads_started":1}
I20260927 06:55:05.608654 7360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d32914ce3024b72a55b8703f765deba" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "30e8b6e93caf4f649f2c446ede4d7957" is_pre_election: true
I20260927 06:55:05.609795 7885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: false } }
I20260927 06:55:05.610180 7360 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9d32914ce3024b72a55b8703f765deba in current term 2: Already voted for candidate 30e8b6e93caf4f649f2c446ede4d7957 in this term.
I20260927 06:55:05.610457 7885 leader_election.cc:288] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [CANDIDATE]: Term 2 election: Requested vote from peers f346c97981314f6c97c333fb51cf4e55 (127.0.97.62:44789), 9d32914ce3024b72a55b8703f765deba (127.0.97.61:33033)
I20260927 06:55:05.610896 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.610500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.0.1:34612)
0927 06:55:05.610643 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.610739 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 06:55:05.611052 7165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30e8b6e93caf4f649f2c446ede4d7957" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d32914ce3024b72a55b8703f765deba"
I20260927 06:55:05.611246 7165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 1 FOLLOWER]: Advancing to term 2
I20260927 06:55:05.611230 7142 leader_election.cc:302] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d32914ce3024b72a55b8703f765deba; no voters: 30e8b6e93caf4f649f2c446ede4d7957, f346c97981314f6c97c333fb51cf4e55
I20260927 06:55:05.611893 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.611560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.0.1:34622)
0927 06:55:05.611733 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.611788 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260927 06:55:05.612038 7333 leader_election.cc:334] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f346c97981314f6c97c333fb51cf4e55 (127.0.97.62:44789): Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
I20260927 06:55:05.613420 7165 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e8b6e93caf4f649f2c446ede4d7957 in term 2.
I20260927 06:55:05.613580 7886 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 06:55:05.613793 7336 leader_election.cc:302] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30e8b6e93caf4f649f2c446ede4d7957, 9d32914ce3024b72a55b8703f765deba; no voters: f346c97981314f6c97c333fb51cf4e55
I20260927 06:55:05.614091 7885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 06:55:05.614466 7885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [term 2 LEADER]: Becoming Leader. State: Replica: 30e8b6e93caf4f649f2c446ede4d7957, State: Running, Role: LEADER
I20260927 06:55:05.614800 7885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: false } }
I20260927 06:55:05.616231 7895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30e8b6e93caf4f649f2c446ede4d7957. Latest consensus state: current_term: 2 leader_uuid: "30e8b6e93caf4f649f2c446ede4d7957" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: false } } }
I20260927 06:55:05.616580 7895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:05.617226 7897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:55:05.620556 7897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:55:05.621335 7897 catalog_manager.cc:1259] Loaded cluster ID: 17f0f544b36d4aad874de996e288df85
I20260927 06:55:05.621439 7897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:55:05.622630 7897 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:55:05.623493 7897 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: Loaded TSK: 0
I20260927 06:55:05.624235 7897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:55:05.654193 7350 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:55:05.660989 7350 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:55:05.664842 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.664487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.0.1:34640)
0927 06:55:05.664637 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.664726 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 06:55:05.664955 7165 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [term 2 FOLLOWER]: Refusing update from remote peer 30e8b6e93caf4f649f2c446ede4d7957: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 06:55:05.665747 7895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 06:55:05.665935 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.665530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.0.1:34654)
0927 06:55:05.665698 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.665795 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 06:55:05.666218 7333 consensus_peers.cc:487] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 -> Peer f346c97981314f6c97c333fb51cf4e55 (127.0.97.62:44789): Couldn't send request to peer f346c97981314f6c97c333fb51cf4e55. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 06:55:05.668089 7886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30e8b6e93caf4f649f2c446ede4d7957. Latest consensus state: current_term: 2 leader_uuid: "30e8b6e93caf4f649f2c446ede4d7957" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: false } } }
I20260927 06:55:05.668361 7886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:55:05.669878 7895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "30e8b6e93caf4f649f2c446ede4d7957" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: false } } }
I20260927 06:55:05.670207 7895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:05.670493 7886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "30e8b6e93caf4f649f2c446ede4d7957" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f346c97981314f6c97c333fb51cf4e55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44789 } } peers { permanent_uuid: "9d32914ce3024b72a55b8703f765deba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33033 } attrs { promote: false } } peers { permanent_uuid: "30e8b6e93caf4f649f2c446ede4d7957" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38017 } attrs { promote: false } } }
I20260927 06:55:05.670747 7886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d32914ce3024b72a55b8703f765deba [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:55:05.670958 7899 mvcc.cc:205] Tried to move back new op lower bound from 7333855664797274112 to 7333855664601284608. Current Snapshot: MvccSnapshot[applied={T|T < 7333855664797274112}]
I20260927 06:55:05.687610 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.677436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33493 (local address 127.0.0.1:33474)
0927 06:55:05.677571 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.677591 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:05.677716 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:05.678967 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:05.678975 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:05.679012 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:05.679279 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:05.679293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.681722 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.681731 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:05.684439 (+ 2708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:05.684459 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.686515 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.686524 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:05.686851 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:05.686921 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 06:55:05.687004 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 06:55:05.687196 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 06:55:05.688962 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.678181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.0.97.2:33493)
0927 06:55:05.678605 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:05.678612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:05.678630 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:55:05.678696 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:05.678701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:05.678768 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:05.679445 (+ 677us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:05.679651 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.681517 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.684699 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.686318 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.688178 (+ 1860us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:05.688318 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:05.688380 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 06:55:05.688570 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":226,"threads_started":1}
I20260927 06:55:05.689615 7605 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe5a0efa89548adbb9463e6d1eb896e (DEFAULT_TABLE table=test-table [id=806555ea548e43e9be7edc4c307541ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:05.690086 7605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe5a0efa89548adbb9463e6d1eb896e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:05.695052 7908 tablet_bootstrap.cc:492] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479: Bootstrap starting.
I20260927 06:55:05.696962 7908 tablet_bootstrap.cc:654] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:05.705178 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.678597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41593 (local address 127.0.0.1:44822)
0927 06:55:05.687834 (+ 9237us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.687850 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:05.687967 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:05.696442 (+ 8475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:05.696447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:05.696481 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:05.696698 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:05.696710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.699482 (+ 2772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.699491 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:05.702227 (+ 2736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:05.702249 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.704346 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.704352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:05.704623 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:05.704662 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 06:55:05.704735 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 06:55:05.704872 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9160,"threads_started":1}
I20260927 06:55:05.705511 7908 tablet_bootstrap.cc:492] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479: No bootstrap required, opened a new log
I20260927 06:55:05.705708 7908 ts_tablet_manager.cc:1397] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260927 06:55:05.705965 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.678704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44822 (local address 127.0.97.3:41593)
0927 06:55:05.695984 (+ 17280us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:05.695993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:05.696017 (+ 24us) server_negotiation.cc:407] Connection header received
0927 06:55:05.696098 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:05.696104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:05.696181 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:05.696852 (+ 671us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:05.697039 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.699280 (+ 2241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.702556 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.704166 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.705378 (+ 1212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:05.705497 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:05.705551 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 06:55:05.705684 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17129,"thread_start_us":17570,"threads_started":1}
I20260927 06:55:05.706774 7908 raft_consensus.cc:348] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } } peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } }
I20260927 06:55:05.707091 7908 raft_consensus.cc:374] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:05.707193 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.678340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36787 (local address 127.0.0.1:46146)
0927 06:55:05.678689 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.678708 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:05.678825 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:05.698982 (+ 20157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:05.698990 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:05.699032 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:05.699484 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:05.699500 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.701377 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.701385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:05.704316 (+ 2931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:05.704336 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.706513 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.706522 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:05.706681 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:05.706718 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 06:55:05.706773 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 06:55:05.706914 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260927 06:55:05.707217 7908 raft_consensus.cc:729] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 151233c285ec4937b55164d138143479, State: Initialized, Role: FOLLOWER
I20260927 06:55:05.707310 7746 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe5a0efa89548adbb9463e6d1eb896e (DEFAULT_TABLE table=test-table [id=806555ea548e43e9be7edc4c307541ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:05.707497 7908 consensus_queue.cc:260] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } } peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } }
I20260927 06:55:05.707738 7746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe5a0efa89548adbb9463e6d1eb896e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:05.707927 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.678409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46146 (local address 127.0.97.1:36787)
0927 06:55:05.698582 (+ 20173us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:05.698591 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:05.698616 (+ 25us) server_negotiation.cc:407] Connection header received
0927 06:55:05.698700 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:05.698708 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:05.698804 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:05.698931 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:05.699814 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.701167 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.704564 (+ 3397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.706276 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.707390 (+ 1114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:05.707475 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:05.707523 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 06:55:05.707652 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20013,"thread_start_us":19873,"threads_started":1}
I20260927 06:55:05.708298 7908 ts_tablet_manager.cc:1428] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 06:55:05.712109 7670 heartbeater.cc:503] Master 127.0.97.60:38017 was elected leader, sending a full tablet report...
I20260927 06:55:05.713459 7912 tablet_bootstrap.cc:492] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f: Bootstrap starting.
I20260927 06:55:05.714099 7465 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe5a0efa89548adbb9463e6d1eb896e (DEFAULT_TABLE table=test-table [id=806555ea548e43e9be7edc4c307541ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:05.714516 7465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe5a0efa89548adbb9463e6d1eb896e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:05.714999 7912 tablet_bootstrap.cc:654] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:05.716917 7912 tablet_bootstrap.cc:492] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f: No bootstrap required, opened a new log
I20260927 06:55:05.717078 7912 ts_tablet_manager.cc:1397] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 06:55:05.717702 7912 raft_consensus.cc:348] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } } peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } }
I20260927 06:55:05.717892 7912 raft_consensus.cc:374] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:05.717999 7912 raft_consensus.cc:729] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eebd909c11f4326bb832093502f823f, State: Initialized, Role: FOLLOWER
I20260927 06:55:05.718273 7912 consensus_queue.cc:260] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } } peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } }
I20260927 06:55:05.719220 7912 ts_tablet_manager.cc:1428] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 06:55:05.722510 7811 heartbeater.cc:503] Master 127.0.97.60:38017 was elected leader, sending a full tablet report...
I20260927 06:55:05.726550 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.708690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.2:35749)
0927 06:55:05.726035 (+ 17345us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.726209 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17184,"threads_started":1}
I20260927 06:55:05.727854 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.727376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.2:52371)
0927 06:55:05.727612 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.727700 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 06:55:05.731016 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.719718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.3:39907)
0927 06:55:05.730512 (+ 10794us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.730673 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10638,"thread_start_us":10497,"threads_started":1}
I20260927 06:55:05.731667 7915 tablet_bootstrap.cc:492] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40: Bootstrap starting.
I20260927 06:55:05.731999 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.731681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.3:45691)
0927 06:55:05.731827 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.731898 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 06:55:05.733060 7915 tablet_bootstrap.cc:654] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:05.734997 7915 tablet_bootstrap.cc:492] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40: No bootstrap required, opened a new log
I20260927 06:55:05.735146 7915 ts_tablet_manager.cc:1397] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 06:55:05.735844 7915 raft_consensus.cc:348] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } } peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } }
I20260927 06:55:05.736022 7915 raft_consensus.cc:374] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:05.736090 7915 raft_consensus.cc:729] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3b556cc4054b7ca08c3abe1a70ae40, State: Initialized, Role: FOLLOWER
I20260927 06:55:05.736289 7915 consensus_queue.cc:260] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } } peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } }
I20260927 06:55:05.736891 7915 ts_tablet_manager.cc:1428] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 06:55:05.738123 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.737204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.1:51083)
0927 06:55:05.737645 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.737834 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":193,"threads_started":1}
I20260927 06:55:05.739439 7530 heartbeater.cc:503] Master 127.0.97.60:38017 was elected leader, sending a full tablet report...
I20260927 06:55:05.739630 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.738667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.1:56045)
0927 06:55:05.739179 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.739317 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":243,"threads_started":1}
W20260927 06:55:05.878481 7533 tablet.cc:2369] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:55:05.980341 7914 raft_consensus.cc:482] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:55:05.980568 7914 raft_consensus.cc:504] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } } peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } }
I20260927 06:55:05.982079 7914 leader_election.cc:288] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 151233c285ec4937b55164d138143479 (127.0.97.2:33493), 8b3b556cc4054b7ca08c3abe1a70ae40 (127.0.97.1:36787)
I20260927 06:55:05.987929 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.982508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52583 (local address 127.0.97.2:33493)
0927 06:55:05.982730 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:05.982736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:05.982989 (+ 253us) server_negotiation.cc:407] Connection header received
0927 06:55:05.983227 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:05.983231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:05.983288 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:05.983381 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:05.983824 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.984507 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.986256 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.986985 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.987121 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:05.987219 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:05.987559 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 06:55:05.987682 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 06:55:05.987983 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.982391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33493 (local address 127.0.97.3:52583)
0927 06:55:05.982857 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.982890 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:05.983082 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:05.983395 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:05.983400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:05.983422 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:05.983684 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:05.983692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.984644 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.984651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:05.986089 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:05.986100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.987162 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.987168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:05.987335 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:05.987405 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 06:55:05.987526 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 06:55:05.987656 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":35}
I20260927 06:55:05.988059 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.982447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59073 (local address 127.0.97.1:36787)
0927 06:55:05.982733 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:05.982739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:05.983185 (+ 446us) server_negotiation.cc:407] Connection header received
0927 06:55:05.983368 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:05.983372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:05.983427 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:05.983515 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:05.984031 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.984702 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.986394 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.987147 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.987295 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:05.987396 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:05.987675 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 06:55:05.987799 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260927 06:55:05.988359 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.982418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36787 (local address 127.0.97.3:59073)
0927 06:55:05.983085 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.983099 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:05.983233 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:05.983541 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:05.983545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:05.983569 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:05.983890 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:05.983897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.984855 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.984865 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:05.986258 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:05.986269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:05.987327 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:05.987333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:05.987526 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:05.987547 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 06:55:05.987653 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 06:55:05.987785 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":4,"thread_start_us":400,"threads_started":1}
I20260927 06:55:05.988680 7624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe5a0efa89548adbb9463e6d1eb896e" candidate_uuid: "9eebd909c11f4326bb832093502f823f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "151233c285ec4937b55164d138143479" is_pre_election: true
I20260927 06:55:05.988981 7624 raft_consensus.cc:2454] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9eebd909c11f4326bb832093502f823f in term 0.
I20260927 06:55:05.989151 7485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe5a0efa89548adbb9463e6d1eb896e" candidate_uuid: "9eebd909c11f4326bb832093502f823f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" is_pre_election: true
I20260927 06:55:05.989485 7700 leader_election.cc:302] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 151233c285ec4937b55164d138143479, 9eebd909c11f4326bb832093502f823f; no voters:
I20260927 06:55:05.989555 7485 raft_consensus.cc:2454] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9eebd909c11f4326bb832093502f823f in term 0.
I20260927 06:55:05.989840 7914 raft_consensus.cc:2781] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:55:05.989985 7914 raft_consensus.cc:482] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:55:05.990113 7914 raft_consensus.cc:3037] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:05.992264 7914 raft_consensus.cc:504] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } } peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } }
I20260927 06:55:05.992888 7914 leader_election.cc:288] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [CANDIDATE]: Term 1 election: Requested vote from peers 151233c285ec4937b55164d138143479 (127.0.97.2:33493), 8b3b556cc4054b7ca08c3abe1a70ae40 (127.0.97.1:36787)
I20260927 06:55:05.993227 7624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe5a0efa89548adbb9463e6d1eb896e" candidate_uuid: "9eebd909c11f4326bb832093502f823f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "151233c285ec4937b55164d138143479"
I20260927 06:55:05.993337 7485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe5a0efa89548adbb9463e6d1eb896e" candidate_uuid: "9eebd909c11f4326bb832093502f823f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40"
I20260927 06:55:05.993440 7624 raft_consensus.cc:3037] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:05.993556 7485 raft_consensus.cc:3037] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:05.995406 7624 raft_consensus.cc:2454] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eebd909c11f4326bb832093502f823f in term 1.
I20260927 06:55:05.995582 7485 raft_consensus.cc:2454] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eebd909c11f4326bb832093502f823f in term 1.
I20260927 06:55:05.995754 7700 leader_election.cc:302] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 151233c285ec4937b55164d138143479, 9eebd909c11f4326bb832093502f823f; no voters:
I20260927 06:55:05.995996 7914 raft_consensus.cc:2781] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:55:05.996395 7914 raft_consensus.cc:686] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [term 1 LEADER]: Becoming Leader. State: Replica: 9eebd909c11f4326bb832093502f823f, State: Running, Role: LEADER
I20260927 06:55:05.996995 7914 consensus_queue.cc:237] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } } peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } }
I20260927 06:55:05.997306 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:05.996905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.3:59321)
0927 06:55:05.997060 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:05.997154 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 06:55:06.000175 7349 catalog_manager.cc:5495] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9eebd909c11f4326bb832093502f823f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "9eebd909c11f4326bb832093502f823f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9eebd909c11f4326bb832093502f823f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } health_report { overall_health: HEALTHY } } }
I20260927 06:55:06.083518 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:06.078870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.0.97.3:41593)
0927 06:55:06.079051 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:06.079055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:06.079072 (+ 17us) server_negotiation.cc:407] Connection header received
0927 06:55:06.079153 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:06.079157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:06.079233 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:06.079352 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:55:06.079663 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:06.080331 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:06.081457 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:06.081899 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:06.081990 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:06.082435 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:55:06.082940 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:55:06.083044 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:06.083146 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 06:55:06.083204 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 06:55:06.087224 7624 raft_consensus.cc:1261] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479 [term 1 FOLLOWER]: Refusing update from remote peer 9eebd909c11f4326bb832093502f823f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:55:06.087489 7485 raft_consensus.cc:1261] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 1 FOLLOWER]: Refusing update from remote peer 9eebd909c11f4326bb832093502f823f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:55:06.087899 7922 consensus_queue.cc:1035] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [LEADER]: Connected to new peer: Peer: permanent_uuid: "151233c285ec4937b55164d138143479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:55:06.088289 7914 consensus_queue.cc:1035] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b3b556cc4054b7ca08c3abe1a70ae40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:55:06.091570 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:06.091078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.1:49303)
0927 06:55:06.091251 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:06.091361 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 06:55:06.092007 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:06.090388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.2:34269)
0927 06:55:06.091702 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:06.091789 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1192}
I20260927 06:55:06.096516 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:06.095632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.3:41571)
0927 06:55:06.096266 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:06.096352 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513}
I20260927 06:55:06.099334 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:06.098847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.3:50717)
0927 06:55:06.098984 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:06.099070 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 06:55:06.099439 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:06.098985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.1:46217)
0927 06:55:06.099185 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:06.099276 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 06:55:06.102965 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:06.102662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.97.2:56515)
0927 06:55:06.102787 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:06.102852 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 06:55:06.110682 7350 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:55:06.110893 7350 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:55:06.115235 7350 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957: Sending DeleteTablet for 3 replicas of tablet 6fe5a0efa89548adbb9463e6d1eb896e
I20260927 06:55:06.116123 7605 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe5a0efa89548adbb9463e6d1eb896e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:06 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:33474
I20260927 06:55:06.116333 7465 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe5a0efa89548adbb9463e6d1eb896e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:06 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:46146
I20260927 06:55:06.116856 7934 tablet_replica.cc:330] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479: stopping tablet replica
I20260927 06:55:06.117100 7934 raft_consensus.cc:2229] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:55:06.117223 7935 tablet_replica.cc:330] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40: stopping tablet replica
I20260927 06:55:06.117780 7746 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe5a0efa89548adbb9463e6d1eb896e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:06 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:44822
I20260927 06:55:06.117791 7934 raft_consensus.cc:2258] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:06.118393 7935 raft_consensus.cc:2229] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:55:06.118685 7935 raft_consensus.cc:2258] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:06.118902 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 7337
I20260927 06:55:06.119771 7935 ts_tablet_manager.cc:1905] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:06.127483 7934 ts_tablet_manager.cc:1905] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:06.128542 7935 ts_tablet_manager.cc:1918] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:55:06.128681 7935 log.cc:1196] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/6fe5a0efa89548adbb9463e6d1eb896e
I20260927 06:55:06.129196 7935 ts_tablet_manager.cc:1939] T 6fe5a0efa89548adbb9463e6d1eb896e P 8b3b556cc4054b7ca08c3abe1a70ae40: Deleting consensus metadata
I20260927 06:55:06.130571 7335 catalog_manager.cc:4841] TS 8b3b556cc4054b7ca08c3abe1a70ae40 (127.0.97.1:36787): tablet 6fe5a0efa89548adbb9463e6d1eb896e (table test-table [id=806555ea548e43e9be7edc4c307541ce]) successfully deleted
I20260927 06:55:06.137203 7934 ts_tablet_manager.cc:1918] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:55:06.137341 7934 log.cc:1196] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/6fe5a0efa89548adbb9463e6d1eb896e
I20260927 06:55:06.137838 7934 ts_tablet_manager.cc:1939] T 6fe5a0efa89548adbb9463e6d1eb896e P 151233c285ec4937b55164d138143479: Deleting consensus metadata
I20260927 06:55:06.139343 7936 tablet_replica.cc:330] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f: stopping tablet replica
I20260927 06:55:06.139585 7936 raft_consensus.cc:2229] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:55:06.140321 7334 catalog_manager.cc:4841] TS 151233c285ec4937b55164d138143479 (127.0.97.2:33493): tablet 6fe5a0efa89548adbb9463e6d1eb896e (table test-table [id=806555ea548e43e9be7edc4c307541ce]) successfully deleted
I20260927 06:55:06.140413 7936 raft_consensus.cc:2258] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:06.141407 7936 ts_tablet_manager.cc:1905] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:06.148331 7936 ts_tablet_manager.cc:1918] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:55:06.148468 7936 log.cc:1196] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/6fe5a0efa89548adbb9463e6d1eb896e
I20260927 06:55:06.149000 7936 ts_tablet_manager.cc:1939] T 6fe5a0efa89548adbb9463e6d1eb896e P 9eebd909c11f4326bb832093502f823f: Deleting consensus metadata
I20260927 06:55:06.150153 7335 catalog_manager.cc:4841] TS 9eebd909c11f4326bb832093502f823f (127.0.97.3:41593): tablet 6fe5a0efa89548adbb9463e6d1eb896e (table test-table [id=806555ea548e43e9be7edc4c307541ce]) successfully deleted
I20260927 06:55:06.153957 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:06.119943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.0.97.1:36787)
0927 06:55:06.120100 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:06.120108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:06.138578 (+ 18470us) server_negotiation.cc:407] Connection header received
0927 06:55:06.138779 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:06.138785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:06.138880 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:06.138978 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:06.140857 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:06.141880 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:06.143656 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:06.144245 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:06.144333 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:06.147834 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:06.147853 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:06.147873 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:06.147910 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:06.151247 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:06.152033 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:06.152042 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:06.152045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:06.152184 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:06.152613 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:06.152616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:06.152618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:06.152989 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:06.153226 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:06.153528 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 06:55:06.153674 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 06:55:06.154680 7525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:06.248701 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:06.247948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.0.1:34656)
0927 06:55:06.248385 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:06.248480 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":212,"threads_started":1}
I20260927 06:55:06.250087 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:06.249661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.0.1:34670)
0927 06:55:06.249830 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:06.249936 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 06:55:06.286952 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 7337
I20260927 06:55:06.309034 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 7542
I20260927 06:55:06.323105 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:06.309722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.0.97.2:33493)
0927 06:55:06.309911 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:06.309918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:06.309941 (+ 23us) server_negotiation.cc:407] Connection header received
0927 06:55:06.310093 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:06.310099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:06.310176 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:06.310282 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:06.311401 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:06.312511 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:06.313662 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:06.314342 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:06.314461 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:06.316776 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:06.316793 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:06.316811 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:06.316847 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:06.320245 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:06.320835 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:06.320846 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:06.320849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:06.321012 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:06.321409 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:06.321415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:06.321418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:06.321835 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:06.322148 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:06.322469 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 06:55:06.322714 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 06:55:06.323760 7665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:06.421455 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 7542
I20260927 06:55:06.441587 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 7683
I20260927 06:55:06.455176 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:06.442565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44836 (local address 127.0.97.3:41593)
0927 06:55:06.442737 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:06.442744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:06.442759 (+ 15us) server_negotiation.cc:407] Connection header received
0927 06:55:06.442814 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:06.442818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:06.442875 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:06.442977 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:06.444256 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:06.445029 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:06.446431 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:06.446851 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:06.447012 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:06.449373 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:06.449384 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:06.449399 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:06.449432 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:06.452446 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:06.453097 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:06.453104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:06.453106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:06.453234 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:06.453602 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:06.453609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:06.453613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:06.453974 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:06.454424 (+ 450us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:06.454631 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 06:55:06.454832 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 06:55:06.455775 7806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:06.580355 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 7683
I20260927 06:55:06.601327 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 7012
I20260927 06:55:06.602370 7185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:06.672457 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:06.672004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.0.1:34674)
0927 06:55:06.672220 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:06.672322 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 06:55:06.673359 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:06.673049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44789 (local address 127.0.0.1:34690)
0927 06:55:06.673182 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:06.673251 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 06:55:06.715683 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 7012
W20260927 06:55:06.736207 7336 connection.cc:535] client connection to 127.0.97.61:33033 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 06:55:06.736570 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 7112
I20260927 06:55:06.737491 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:06.736975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33033 (local address 127.0.0.1:32890)
0927 06:55:06.737160 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:06.737303 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 06:55:06.737519 7380 generic_service.cc:196] Checking for leaks (request via RPC)
W20260927 06:55:06.737701 7336 consensus_peers.cc:487] T 00000000000000000000000000000000 P 30e8b6e93caf4f649f2c446ede4d7957 -> Peer 9d32914ce3024b72a55b8703f765deba (127.0.97.61:33033): Couldn't send request to peer 9d32914ce3024b72a55b8703f765deba. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 06:55:06.853788 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 7112
2026-09-27T06:55:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9456 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:55:06 dist-test-slave-f892 krb5kdc[7947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:55:06 dist-test-slave-f892 krb5kdc[7947](info): set up 2 sockets
Sep 27 06:55:06 dist-test-slave-f892 krb5kdc[7947](info): commencing operation
krb5kdc: starting...
W20260927 06:55:07.938838 388 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:55:08 dist-test-slave-f892 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:55:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:55:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260927 06:55:08.139626 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41405
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:55:08.447230 7960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:08.447489 7960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:08.447573 7960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:08.455348 7960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:55:08.455454 7960 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 06:55:08.455498 7960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:08.455567 7960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:08.455619 7960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:55:08.455688 7960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:55:08.465610 7960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41405
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:08.466746 7960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:08.467954 7960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:08.479032 7968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:08.479115 7965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:08.479084 7966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:08.479794 7960 server_base.cc:1034] running on GCE node
I20260927 06:55:08.480417 7960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:08.481668 7960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:08.482899 7960 hybrid_clock.cc:648] HybridClock initialized: now 1790492108482816 us; error 47 us; skew 500 ppm
I20260927 06:55:08.483232 7960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:08 dist-test-slave-f892 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492108, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:08.486244 7960 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260927 06:55:08.487507 7960 webserver.cc:460] Webserver started at http://127.0.97.62:40187/ using document root <none> and password file <none>
I20260927 06:55:08.488103 7960 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:08.488180 7960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:08.488433 7960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:08.491150 7960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "3dc9badb81654bed99e411125dbe582a"
format_stamp: "Formatted at 2026-09-27 06:55:08 on dist-test-slave-f892"
server_key: "5163a46acbba3c29c7de0edf318a5394"
server_key_iv: "9063c64e57032c20d57869e1eb08f981"
server_key_version: "encryptionkey@0"
I20260927 06:55:08.491815 7960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "3dc9badb81654bed99e411125dbe582a"
format_stamp: "Formatted at 2026-09-27 06:55:08 on dist-test-slave-f892"
server_key: "5163a46acbba3c29c7de0edf318a5394"
server_key_iv: "9063c64e57032c20d57869e1eb08f981"
server_key_version: "encryptionkey@0"
I20260927 06:55:08.496421 7960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 06:55:08.499593 7975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:08.500735 7960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 06:55:08.500903 7960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "3dc9badb81654bed99e411125dbe582a"
format_stamp: "Formatted at 2026-09-27 06:55:08 on dist-test-slave-f892"
server_key: "5163a46acbba3c29c7de0edf318a5394"
server_key_iv: "9063c64e57032c20d57869e1eb08f981"
server_key_version: "encryptionkey@0"
I20260927 06:55:08.501106 7960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:08.518196 7960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:08.522311 7960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:08.522565 7960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:08.542788 7960 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41405
I20260927 06:55:08.542806 8036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41405 every 8 connection(s)
I20260927 06:55:08.544014 7960 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:55:08.547564 8037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:08.547971 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 7960
I20260927 06:55:08.548166 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:55:08.548966 388 external_mini_cluster.cc:1428] Setting key 7b498e40e1901603edf424f51ba079be
I20260927 06:55:08.561090 8037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a: Bootstrap starting.
Sep 27 06:55:08 dist-test-slave-f892 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 06:55:08.564586 8037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:08.565619 8037 log.cc:824] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a: Log is configured to *not* fsync() on all Append() calls
I20260927 06:55:08.568650 8037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a: No bootstrap required, opened a new log
I20260927 06:55:08.571712 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:08.550909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39362 (local address 127.0.97.62:41405)
0927 06:55:08.551678 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:08.551688 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:08.551728 (+ 40us) server_negotiation.cc:407] Connection header received
0927 06:55:08.553690 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:08.553710 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:08.554144 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:08.554451 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:08.555996 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:08.557222 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:08.558993 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:08.559593 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:08.559710 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:08.564083 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:08.564091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:08.564104 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:08.564136 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:08.567567 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:08.568177 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:08.568184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:08.568193 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:08.568402 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:08.568741 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:08.568746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:08.568749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:08.569400 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:08.569688 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:08.569992 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 06:55:08.570421 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":347,"threads_started":1}
I20260927 06:55:08.577319 8037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc9badb81654bed99e411125dbe582a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41405 } }
I20260927 06:55:08.577661 8037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:08.577735 8037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc9badb81654bed99e411125dbe582a, State: Initialized, Role: FOLLOWER
I20260927 06:55:08.578222 8037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc9badb81654bed99e411125dbe582a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41405 } }
I20260927 06:55:08.578421 8037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:55:08.578510 8037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:55:08.578622 8037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:08.580528 8037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc9badb81654bed99e411125dbe582a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41405 } }
I20260927 06:55:08.580921 8037 leader_election.cc:302] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dc9badb81654bed99e411125dbe582a; no voters:
I20260927 06:55:08.581398 8037 leader_election.cc:288] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:55:08.581532 8042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:55:08.582229 8042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a [term 1 LEADER]: Becoming Leader. State: Replica: 3dc9badb81654bed99e411125dbe582a, State: Running, Role: LEADER
I20260927 06:55:08.582719 8042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc9badb81654bed99e411125dbe582a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41405 } }
I20260927 06:55:08.583385 8037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:55:08.585264 8044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dc9badb81654bed99e411125dbe582a. Latest consensus state: current_term: 1 leader_uuid: "3dc9badb81654bed99e411125dbe582a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc9badb81654bed99e411125dbe582a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41405 } } }
I20260927 06:55:08.585695 8044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:08.585301 8043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dc9badb81654bed99e411125dbe582a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc9badb81654bed99e411125dbe582a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41405 } } }
I20260927 06:55:08.586421 8043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:08.586506 8048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:55:08.591140 8048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:55:08.602145 8048 catalog_manager.cc:1347] Generated new cluster ID: ea4fca375c944cb58fd62e4f1864a061
I20260927 06:55:08.602244 8048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:55:08.628345 8048 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:55:08.629601 8048 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:55:08.635129 8048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a: Generated new TSK 0
I20260927 06:55:08.635958 8048 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260927 06:55:08.700824 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41405
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:55:09.055511 8063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:09.055756 8063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:09.055915 8063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:09.063104 8063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:09.063270 8063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:09.063432 8063 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 06:55:09.063546 8063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:55:09.073637 8063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:09.074821 8063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:09.076064 8063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:09.088392 8069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:09.088465 8068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:09.088438 8071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:09.089490 8063 server_base.cc:1034] running on GCE node
I20260927 06:55:09.090188 8063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:09.091060 8063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:09.092319 8063 hybrid_clock.cc:648] HybridClock initialized: now 1790492109092262 us; error 35 us; skew 500 ppm
I20260927 06:55:09.092602 8063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:09 dist-test-slave-f892 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492109, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:09.095688 8063 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 06:55:09.097040 8063 webserver.cc:460] Webserver started at http://127.0.97.1:46629/ using document root <none> and password file <none>
I20260927 06:55:09.097651 8063 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:09.097731 8063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:09.097980 8063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:09.100862 8063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "366565fc5a9a4d3d806537f3bd3d9afe"
format_stamp: "Formatted at 2026-09-27 06:55:09 on dist-test-slave-f892"
server_key: "dbee0e1609749c4725cb85a4524fdeda"
server_key_iv: "27e8fd55cccf52f9d68aef68c3652ef4"
server_key_version: "encryptionkey@0"
I20260927 06:55:09.101558 8063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "366565fc5a9a4d3d806537f3bd3d9afe"
format_stamp: "Formatted at 2026-09-27 06:55:09 on dist-test-slave-f892"
server_key: "dbee0e1609749c4725cb85a4524fdeda"
server_key_iv: "27e8fd55cccf52f9d68aef68c3652ef4"
server_key_version: "encryptionkey@0"
I20260927 06:55:09.106323 8063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260927 06:55:09.109339 8078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:09.110579 8063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 06:55:09.110702 8063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "366565fc5a9a4d3d806537f3bd3d9afe"
format_stamp: "Formatted at 2026-09-27 06:55:09 on dist-test-slave-f892"
server_key: "dbee0e1609749c4725cb85a4524fdeda"
server_key_iv: "27e8fd55cccf52f9d68aef68c3652ef4"
server_key_version: "encryptionkey@0"
I20260927 06:55:09.110862 8063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:09.135515 8063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:09.139524 8063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:09.139735 8063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:09.142144 8063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:09.142221 8063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:09.142380 8063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:09.142469 8063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 06:55:09 dist-test-slave-f892 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492109, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 06:55:09.163599 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:09.142632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42783 (local address 127.0.97.62:41405)
0927 06:55:09.143290 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:09.143296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:09.144327 (+ 1031us) server_negotiation.cc:407] Connection header received
0927 06:55:09.146322 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:09.146327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:09.146431 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:09.146542 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:09.148485 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:09.149451 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:09.152578 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:09.153081 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:09.153239 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:09.157760 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:09.157765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:09.157768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:09.157799 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:09.160686 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:09.161354 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:09.161360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:09.161371 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:09.161487 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:09.161932 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:09.161937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:09.161940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:09.162247 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:09.162402 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:09.163036 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 06:55:09.163247 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":440,"threads_started":1}
I20260927 06:55:09.164623 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:09.143321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41405 (local address 127.0.97.1:42783)
0927 06:55:09.144168 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:09.144213 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:09.146074 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:09.146787 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:09.146801 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:09.147575 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:09.148319 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:09.148340 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:09.149623 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:09.149636 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:09.152402 (+ 2766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:09.152418 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:09.153356 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:09.153367 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:09.153734 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:09.154676 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:09.154722 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:09.157547 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:09.160815 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:09.160824 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:09.160838 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:09.161231 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:09.161603 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:09.161607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:09.161611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:09.161819 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:09.162370 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:09.162378 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:09.162694 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 06:55:09.162934 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 06:55:09.163397 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":347,"threads_started":1}
I20260927 06:55:09.183807 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:09.170587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41405 (local address 127.0.97.1:43651)
0927 06:55:09.171234 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:09.171255 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:09.171388 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:09.171704 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:09.171708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:09.172137 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:09.172411 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:09.172420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:09.173366 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:09.173372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:09.174587 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:09.174606 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:09.175592 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:09.175598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:09.175760 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:09.176478 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:09.176494 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:09.177316 (+ 822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:09.180916 (+ 3600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:09.180920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:09.180922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:09.181482 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:09.181892 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:09.181900 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:09.181907 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:09.182142 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:09.182940 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:09.182947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:09.183103 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 06:55:09.183238 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 06:55:09.183475 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":380,"threads_started":1}
I20260927 06:55:09.184165 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:09.170358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43651 (local address 127.0.97.62:41405)
0927 06:55:09.170505 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:09.170509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:09.171330 (+ 821us) server_negotiation.cc:407] Connection header received
0927 06:55:09.171512 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:09.171518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:09.171589 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:09.171685 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:09.172545 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:09.173230 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:09.174785 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:09.175388 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:09.175517 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:09.177560 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:09.177564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:09.177567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:09.177594 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:09.180752 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:09.181593 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:09.181601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:09.181607 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:09.181752 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:09.182326 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:09.182334 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:09.182340 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:09.182777 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:09.182989 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:09.183369 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 06:55:09.183643 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 06:55:09.194495 8063 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34427
I20260927 06:55:09.194532 8198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34427 every 8 connection(s)
I20260927 06:55:09.195609 8063 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:55:09.197440 8199 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41405
I20260927 06:55:09.197620 8199 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:09.198256 8199 heartbeater.cc:511] Master 127.0.97.62:41405 requested a full tablet report, sending...
I20260927 06:55:09.199996 7992 ts_manager.cc:194] Registered new tserver with Master: 366565fc5a9a4d3d806537f3bd3d9afe (127.0.97.1:34427)
I20260927 06:55:09.201399 7992 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42783
I20260927 06:55:09.202775 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 8063
I20260927 06:55:09.202966 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:55:09.203550 388 external_mini_cluster.cc:1428] Setting key f1c4243c235eb66d0fe1af8e7865f4f0
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260927 06:55:09.245327 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41405
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:55:09.556545 8203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:09.556785 8203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:09.556886 8203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:09.563917 8203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:09.564011 8203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:09.564090 8203 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 06:55:09.564167 8203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:55:09.573535 8203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:09.574721 8203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:09.575912 8203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:09.588223 8208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:09.588065 8209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:09.588188 8211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:09.589033 8203 server_base.cc:1034] running on GCE node
I20260927 06:55:09.589630 8203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:09.590469 8203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:09.591724 8203 hybrid_clock.cc:648] HybridClock initialized: now 1790492109591663 us; error 49 us; skew 500 ppm
I20260927 06:55:09.592001 8203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:09 dist-test-slave-f892 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492109, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:09.594945 8203 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 06:55:09.596256 8203 webserver.cc:460] Webserver started at http://127.0.97.2:44087/ using document root <none> and password file <none>
I20260927 06:55:09.596894 8203 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:09.597013 8203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:09.597311 8203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:09.599996 8203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "df994aaed44b4eda91d639dde8b5528f"
format_stamp: "Formatted at 2026-09-27 06:55:09 on dist-test-slave-f892"
server_key: "da9cb54291e392dc31b4f5c25e21c7f2"
server_key_iv: "c0b652fb80f50976e7559b8a53eb1347"
server_key_version: "encryptionkey@0"
I20260927 06:55:09.600770 8203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "df994aaed44b4eda91d639dde8b5528f"
format_stamp: "Formatted at 2026-09-27 06:55:09 on dist-test-slave-f892"
server_key: "da9cb54291e392dc31b4f5c25e21c7f2"
server_key_iv: "c0b652fb80f50976e7559b8a53eb1347"
server_key_version: "encryptionkey@0"
I20260927 06:55:09.606144 8203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 06:55:09.609284 8218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:09.610550 8203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:55:09.610708 8203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "df994aaed44b4eda91d639dde8b5528f"
format_stamp: "Formatted at 2026-09-27 06:55:09 on dist-test-slave-f892"
server_key: "da9cb54291e392dc31b4f5c25e21c7f2"
server_key_iv: "c0b652fb80f50976e7559b8a53eb1347"
server_key_version: "encryptionkey@0"
I20260927 06:55:09.610912 8203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:09.625459 8203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:09.629482 8203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:09.629746 8203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:09.632282 8203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:09.632388 8203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:09.632568 8203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:09.632651 8203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 06:55:09 dist-test-slave-f892 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492109, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 06:55:09.653925 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:09.632760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37079 (local address 127.0.97.62:41405)
0927 06:55:09.632945 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:09.632951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:09.634317 (+ 1366us) server_negotiation.cc:407] Connection header received
0927 06:55:09.636240 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:09.636246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:09.636333 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:09.636437 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:09.638484 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:09.639512 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:09.642199 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:09.642629 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:09.642754 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:09.646834 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:09.646838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:09.646841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:09.646873 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:09.650076 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:09.650881 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:09.650887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:09.650891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:09.651016 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:09.651584 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:09.651589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:09.651597 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:09.651997 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:09.652138 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:09.653102 (+ 964us) server_negotiation.cc:299] Negotiation successful
0927 06:55:09.653413 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 06:55:09.654539 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:09.633233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41405 (local address 127.0.97.2:37079)
0927 06:55:09.634131 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:09.634184 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:09.635973 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:09.636653 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:09.636661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:09.637356 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:09.638293 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:09.638319 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:09.639655 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:09.639663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:09.642043 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:09.642064 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:09.642871 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:09.642878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:09.643102 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:09.643910 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:09.643930 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:09.646564 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:09.650247 (+ 3683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:09.650265 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:09.650284 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:09.650756 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:09.651176 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:09.651184 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:09.651188 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:09.651463 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:09.652137 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:09.652145 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:09.652635 (+ 490us) client_negotiation.cc:769] Sending connection context
0927 06:55:09.652951 (+ 316us) client_negotiation.cc:240] Negotiation successful
0927 06:55:09.653409 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":329,"threads_started":1}
I20260927 06:55:09.672593 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:09.660292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52821 (local address 127.0.97.62:41405)
0927 06:55:09.660499 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:09.660507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:09.661154 (+ 647us) server_negotiation.cc:407] Connection header received
0927 06:55:09.661334 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:09.661337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:09.661391 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:09.661466 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:09.663166 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:09.663898 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:09.665184 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:09.665582 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:09.665734 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:09.667570 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:09.667575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:09.667579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:09.667606 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:09.670060 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:09.670734 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:09.670741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:09.670745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:09.670858 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:09.671181 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:09.671187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:09.671191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:09.671496 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:09.671640 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:09.671996 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 06:55:09.672234 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 06:55:09.672724 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:09.660284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41405 (local address 127.0.97.2:52821)
0927 06:55:09.661024 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:09.661052 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:09.661215 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:09.661528 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:09.661535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:09.662317 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:09.662929 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:09.662949 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:09.664026 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:09.664032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:09.665056 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:09.665067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:09.665765 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:09.665771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:09.665928 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:09.666643 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:09.666659 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:09.667413 (+ 754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:09.670191 (+ 2778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:09.670195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:09.670203 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:09.670596 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:09.670958 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:09.670964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:09.670968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:09.671086 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:09.671604 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:09.671609 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:09.671779 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 06:55:09.671913 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 06:55:09.672234 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":378,"threads_started":1}
I20260927 06:55:09.687623 8203 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35641
I20260927 06:55:09.687644 8337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35641 every 8 connection(s)
I20260927 06:55:09.688789 8203 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:55:09.690735 8338 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41405
I20260927 06:55:09.690964 8338 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:09.691562 8338 heartbeater.cc:511] Master 127.0.97.62:41405 requested a full tablet report, sending...
I20260927 06:55:09.692852 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 8203
I20260927 06:55:09.692974 7992 ts_manager.cc:194] Registered new tserver with Master: df994aaed44b4eda91d639dde8b5528f (127.0.97.2:35641)
I20260927 06:55:09.693027 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:55:09.693545 388 external_mini_cluster.cc:1428] Setting key f0b69f68bbc9b8f61b9edfe8740bedd8
I20260927 06:55:09.694119 7992 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37079
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 06:55:09.735380 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41405
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:55:10.053642 8342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:10.053912 8342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:10.054095 8342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:10.061098 8342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:10.061187 8342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:10.061384 8342 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 06:55:10.061502 8342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:55:10.070675 8342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:10.071986 8342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:10.073369 8342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:10.086102 8347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:10.086201 8348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:10.086664 8350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:10.087625 8342 server_base.cc:1034] running on GCE node
I20260927 06:55:10.088236 8342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:10.089094 8342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:10.090312 8342 hybrid_clock.cc:648] HybridClock initialized: now 1790492110090235 us; error 60 us; skew 500 ppm
I20260927 06:55:10.090600 8342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:10 dist-test-slave-f892 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492110, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:10.093515 8342 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 06:55:10.094792 8342 webserver.cc:460] Webserver started at http://127.0.97.3:38017/ using document root <none> and password file <none>
I20260927 06:55:10.095412 8342 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:10.095546 8342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:10.095863 8342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:10.098670 8342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "deb2661436744881b57bddbd30a1950a"
format_stamp: "Formatted at 2026-09-27 06:55:10 on dist-test-slave-f892"
server_key: "2d577e974b77e7bb880463bb5e2042ae"
server_key_iv: "c94ac504b1cda17fb90f8714ce86b798"
server_key_version: "encryptionkey@0"
I20260927 06:55:10.099416 8342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "deb2661436744881b57bddbd30a1950a"
format_stamp: "Formatted at 2026-09-27 06:55:10 on dist-test-slave-f892"
server_key: "2d577e974b77e7bb880463bb5e2042ae"
server_key_iv: "c94ac504b1cda17fb90f8714ce86b798"
server_key_version: "encryptionkey@0"
I20260927 06:55:10.104176 8342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 06:55:10.107416 8357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:10.108589 8342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 06:55:10.108750 8342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "deb2661436744881b57bddbd30a1950a"
format_stamp: "Formatted at 2026-09-27 06:55:10 on dist-test-slave-f892"
server_key: "2d577e974b77e7bb880463bb5e2042ae"
server_key_iv: "c94ac504b1cda17fb90f8714ce86b798"
server_key_version: "encryptionkey@0"
I20260927 06:55:10.108955 8342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:10.140532 8342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:10.144543 8342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:10.144815 8342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:10.147444 8342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:10.147543 8342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:10.147683 8342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:10.147758 8342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 06:55:10 dist-test-slave-f892 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492110, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 06:55:10.169965 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:10.147993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45167 (local address 127.0.97.62:41405)
0927 06:55:10.148148 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:10.148152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:10.149565 (+ 1413us) server_negotiation.cc:407] Connection header received
0927 06:55:10.151840 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:10.151849 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:10.151939 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:10.152013 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:10.154436 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.155542 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.157793 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.158420 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.158549 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:10.163063 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:10.163069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:10.163074 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:10.163115 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:10.165970 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:10.167058 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:10.167064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:10.167070 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:10.167248 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:10.167907 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:10.167913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:10.167916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:10.168213 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:10.168383 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:10.169390 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0927 06:55:10.169606 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 06:55:10.171545 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:10.148564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41405 (local address 127.0.97.3:45167)
0927 06:55:10.149398 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:10.149443 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:10.151514 (+ 2071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:10.152597 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:10.152609 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:10.153346 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:10.154239 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:10.154295 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.155713 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.155717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:10.157655 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:10.157666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.158637 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.158644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:10.158856 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:10.159617 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:10.159653 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:10.162817 (+ 3164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:10.166142 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:10.166151 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:10.166170 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:10.166929 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:10.167406 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:10.167413 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:10.167419 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:10.167749 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:10.168474 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:10.168487 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:10.168920 (+ 433us) client_negotiation.cc:769] Sending connection context
0927 06:55:10.169286 (+ 366us) client_negotiation.cc:240] Negotiation successful
0927 06:55:10.169855 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":406,"threads_started":1}
I20260927 06:55:10.194826 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:10.179164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41405 (local address 127.0.97.3:47695)
0927 06:55:10.179916 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:10.179946 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:10.180158 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:10.180530 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:10.180536 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:10.181312 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:10.181845 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:10.181861 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.182975 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.182983 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:10.184890 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:10.184908 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.185663 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.185673 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:10.185926 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:10.186733 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:10.186765 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:10.187883 (+ 1118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:10.192155 (+ 4272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:10.192159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:10.192162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:10.192523 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:10.192927 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:10.192929 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:10.192932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:10.193036 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:10.193700 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:10.193703 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:10.193896 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 06:55:10.194069 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 06:55:10.194399 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":381,"threads_started":1}
I20260927 06:55:10.195003 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:10.179212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47695 (local address 127.0.97.62:41405)
0927 06:55:10.179426 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:10.179434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:10.180028 (+ 594us) server_negotiation.cc:407] Connection header received
0927 06:55:10.180316 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:10.180323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:10.180393 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:10.180469 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:10.182000 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.182869 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.185038 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.185467 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.185563 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:10.188102 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:10.188108 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:10.188112 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:10.188154 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:10.192019 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:10.192629 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:10.192637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:10.192644 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:10.192819 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:10.193131 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:10.193138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:10.193144 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:10.193582 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:10.193782 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:10.194216 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 06:55:10.194520 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 06:55:10.200954 8342 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39213
I20260927 06:55:10.200975 8476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39213 every 8 connection(s)
I20260927 06:55:10.202100 8342 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:55:10.203776 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 8342
I20260927 06:55:10.203817 8199 heartbeater.cc:503] Master 127.0.97.62:41405 was elected leader, sending a full tablet report...
I20260927 06:55:10.203945 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:55:10.203979 8477 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41405
I20260927 06:55:10.204178 8477 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:10.204526 388 external_mini_cluster.cc:1428] Setting key 077d54bd615dcd91a22e4991740a6884
I20260927 06:55:10.204927 8477 heartbeater.cc:511] Master 127.0.97.62:41405 requested a full tablet report, sending...
I20260927 06:55:10.206234 7992 ts_manager.cc:194] Registered new tserver with Master: deb2661436744881b57bddbd30a1950a (127.0.97.3:39213)
I20260927 06:55:10.206964 7992 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45167
I20260927 06:55:10.208161 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 06:55:10 dist-test-slave-f892 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492110, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 06:55:10 dist-test-slave-f892 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:10 dist-test-slave-f892 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 06:55:10.253626 8087 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:10.243595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33508 (local address 127.0.97.62:41405)
0927 06:55:10.243821 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:10.243827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:10.244079 (+ 252us) server_negotiation.cc:407] Connection header received
0927 06:55:10.244284 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:10.244289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:10.244356 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:10.244463 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:10.245704 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.246537 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.247717 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.248134 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.248259 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:10.253291 (+ 5032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33508: BlockingRecv error: recv got EOF from 127.0.0.1:33508 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 06:55:10 dist-test-slave-f892 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492110, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 06:55:10.268810 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:10.256672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.0.97.62:41405)
0927 06:55:10.256863 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:10.256872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:10.257127 (+ 255us) server_negotiation.cc:407] Connection header received
0927 06:55:10.257326 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:10.257329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:10.257392 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:10.257489 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:10.258607 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.259286 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.260499 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.260912 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.261026 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:10.264263 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:10.264268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:10.264271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:10.264299 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:10.266556 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:10.267122 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:10.267128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:10.267132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:10.267242 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:10.267578 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:10.267584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:10.267587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:10.267834 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:10.268072 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:10.268344 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 06:55:10.268537 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 06:55:10.272442 7991 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:55:10.278934 7991 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:55:10.300926 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:10.290062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.0.97.1:34427)
0927 06:55:10.291040 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:10.291050 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:10.291079 (+ 29us) server_negotiation.cc:407] Connection header received
0927 06:55:10.291231 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:10.291241 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:10.291502 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:10.291698 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:10.292511 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.294820 (+ 2309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.298438 (+ 3618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.299850 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.299998 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:10.300188 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:10.300452 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 06:55:10.300614 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":1113,"threads_started":1}
I20260927 06:55:10.300966 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:10.289806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34427 (local address 127.0.0.1:44110)
0927 06:55:10.290632 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:10.290685 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:10.290867 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:10.291918 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:10.291922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:10.291944 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:10.292299 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:10.292315 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.295039 (+ 2724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.295047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:10.298202 (+ 3155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:10.298225 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.300027 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.300032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:10.300196 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:10.300256 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 06:55:10.300386 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 06:55:10.300561 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":1,"thread_start_us":344,"threads_started":1}
I20260927 06:55:10.302732 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:10.291037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39213 (local address 127.0.0.1:50386)
0927 06:55:10.291638 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:10.291656 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:10.291750 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:10.293787 (+ 2037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:10.293795 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:10.293829 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:10.294403 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:10.294418 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.297036 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.297044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:10.299818 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:10.299834 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.301974 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.301980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:10.302175 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:10.302207 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 06:55:10.302292 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 06:55:10.302416 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":337,"threads_started":1}
I20260927 06:55:10.303544 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:10.291257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.0.97.3:39213)
0927 06:55:10.292802 (+ 1545us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:10.292809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:10.292841 (+ 32us) server_negotiation.cc:407] Connection header received
0927 06:55:10.292925 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:10.292933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:10.293177 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:10.293411 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:10.294650 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.296831 (+ 2181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.300294 (+ 3463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.301793 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.301911 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:10.302135 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:10.303210 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0927 06:55:10.303327 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1386,"thread_start_us":1210,"threads_started":1}
I20260927 06:55:10.305006 8129 tablet_service.cc:1467] Processing CreateTablet for tablet 014d2a6cb6fa46299ec2173069089a87 (DEFAULT_TABLE table=test-table [id=ce438088505c47fa9c1f0ba0e2dcb5fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:10.306349 8129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014d2a6cb6fa46299ec2173069089a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:10.306555 8407 tablet_service.cc:1467] Processing CreateTablet for tablet 014d2a6cb6fa46299ec2173069089a87 (DEFAULT_TABLE table=test-table [id=ce438088505c47fa9c1f0ba0e2dcb5fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:10.307531 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:10.292439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35641 (local address 127.0.0.1:49202)
0927 06:55:10.299023 (+ 6584us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:10.299042 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:10.299146 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:10.301114 (+ 1968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:10.301121 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:10.301149 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:10.301495 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:10.301507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.303203 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.303211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:10.305146 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:10.305159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.306821 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.306827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:10.306987 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:10.307016 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 06:55:10.307111 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 06:55:10.307229 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6468,"thread_start_us":6322,"threads_started":1}
I20260927 06:55:10.307900 8407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014d2a6cb6fa46299ec2173069089a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:10.307929 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:10.292885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49202 (local address 127.0.97.2:35641)
0927 06:55:10.300343 (+ 7458us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:10.300374 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:10.300401 (+ 27us) server_negotiation.cc:407] Connection header received
0927 06:55:10.300492 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:10.300498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:10.300699 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:10.300885 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:10.301686 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.303077 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.305340 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.306623 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.306755 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:10.306931 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:10.307166 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 06:55:10.307317 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7330,"thread_start_us":7207,"threads_started":1}
I20260927 06:55:10.311123 8268 tablet_service.cc:1467] Processing CreateTablet for tablet 014d2a6cb6fa46299ec2173069089a87 (DEFAULT_TABLE table=test-table [id=ce438088505c47fa9c1f0ba0e2dcb5fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:10.312335 8268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014d2a6cb6fa46299ec2173069089a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:10.319674 8501 tablet_bootstrap.cc:492] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a: Bootstrap starting.
I20260927 06:55:10.322365 8501 tablet_bootstrap.cc:654] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:10.322654 8502 tablet_bootstrap.cc:492] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe: Bootstrap starting.
I20260927 06:55:10.323662 8501 log.cc:824] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a: Log is configured to *not* fsync() on all Append() calls
I20260927 06:55:10.325937 8502 tablet_bootstrap.cc:654] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:10.326759 8501 tablet_bootstrap.cc:492] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a: No bootstrap required, opened a new log
I20260927 06:55:10.327028 8501 ts_tablet_manager.cc:1397] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260927 06:55:10.327209 8502 log.cc:824] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe: Log is configured to *not* fsync() on all Append() calls
I20260927 06:55:10.330076 8502 tablet_bootstrap.cc:492] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe: No bootstrap required, opened a new log
I20260927 06:55:10.330081 8504 tablet_bootstrap.cc:492] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f: Bootstrap starting.
I20260927 06:55:10.330314 8502 ts_tablet_manager.cc:1397] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 06:55:10.332701 8504 tablet_bootstrap.cc:654] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:10.333571 8504 log.cc:824] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f: Log is configured to *not* fsync() on all Append() calls
I20260927 06:55:10.337301 8501 raft_consensus.cc:348] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } }
I20260927 06:55:10.337713 8501 raft_consensus.cc:374] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:10.337870 8501 raft_consensus.cc:729] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deb2661436744881b57bddbd30a1950a, State: Initialized, Role: FOLLOWER
I20260927 06:55:10.338546 8501 consensus_queue.cc:260] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } }
I20260927 06:55:10.339602 8477 heartbeater.cc:503] Master 127.0.97.62:41405 was elected leader, sending a full tablet report...
I20260927 06:55:10.340262 8501 ts_tablet_manager.cc:1428] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260927 06:55:10.340420 8502 raft_consensus.cc:348] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } }
I20260927 06:55:10.340822 8502 raft_consensus.cc:374] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:10.340940 8504 tablet_bootstrap.cc:492] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f: No bootstrap required, opened a new log
I20260927 06:55:10.340957 8502 raft_consensus.cc:729] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 366565fc5a9a4d3d806537f3bd3d9afe, State: Initialized, Role: FOLLOWER
I20260927 06:55:10.341214 8504 ts_tablet_manager.cc:1397] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260927 06:55:10.341611 8502 consensus_queue.cc:260] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } }
I20260927 06:55:10.346513 8502 ts_tablet_manager.cc:1428] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260927 06:55:10.350489 8504 raft_consensus.cc:348] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } }
I20260927 06:55:10.350800 8504 raft_consensus.cc:374] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:10.350914 8504 raft_consensus.cc:729] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df994aaed44b4eda91d639dde8b5528f, State: Initialized, Role: FOLLOWER
I20260927 06:55:10.351374 8504 consensus_queue.cc:260] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } }
I20260927 06:55:10.352205 8338 heartbeater.cc:503] Master 127.0.97.62:41405 was elected leader, sending a full tablet report...
I20260927 06:55:10.352631 8504 ts_tablet_manager.cc:1428] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20260927 06:55:10.440692 8339 tablet.cc:2369] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:55:10.447779 8200 tablet.cc:2369] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:55:10.453493 8478 tablet.cc:2369] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:55:10.566159 8509 raft_consensus.cc:482] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:55:10.566520 8509 raft_consensus.cc:504] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } }
I20260927 06:55:10.568166 8509 leader_election.cc:288] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 366565fc5a9a4d3d806537f3bd3d9afe (127.0.97.1:34427), deb2661436744881b57bddbd30a1950a (127.0.97.3:39213)
I20260927 06:55:10.576783 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:10.568705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39521 (local address 127.0.97.1:34427)
0927 06:55:10.568943 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:10.568952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:10.569564 (+ 612us) server_negotiation.cc:407] Connection header received
0927 06:55:10.569708 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:10.569714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:10.569799 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:10.569898 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:10.570568 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.571618 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.574537 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.575746 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.575871 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:10.575971 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:10.576440 (+ 469us) server_negotiation.cc:299] Negotiation successful
0927 06:55:10.576554 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 06:55:10.576786 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:10.568593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49721 (local address 127.0.97.3:39213)
0927 06:55:10.568935 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:10.568943 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:10.569309 (+ 366us) server_negotiation.cc:407] Connection header received
0927 06:55:10.569519 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:10.569538 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:10.569639 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:10.569741 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:10.570514 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.571617 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.574538 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.575746 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.575877 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:10.575971 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:10.576440 (+ 469us) server_negotiation.cc:299] Negotiation successful
0927 06:55:10.576554 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260927 06:55:10.577040 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:10.568534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34427 (local address 127.0.97.2:39521)
0927 06:55:10.569466 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:10.569484 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:10.569593 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:10.569924 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:10.569929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:10.569965 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:10.570427 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:10.570439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.571753 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.571759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:10.574391 (+ 2632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:10.574405 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.576001 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.576007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:10.576262 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:10.576320 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 06:55:10.576407 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 06:55:10.576549 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":134,"thread_start_us":572,"threads_started":1}
I20260927 06:55:10.577086 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:10.568447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39213 (local address 127.0.97.2:49721)
0927 06:55:10.569169 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:10.569190 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:10.569370 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:10.569771 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:10.569775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:10.569793 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:10.570344 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:10.570362 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.571759 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.571763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:10.574392 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:10.574405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.576012 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.576018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:10.576257 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:10.576322 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 06:55:10.576407 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 06:55:10.576549 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":229,"spinlock_wait_cycles":21760,"thread_start_us":515,"threads_started":1}
I20260927 06:55:10.578241 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014d2a6cb6fa46299ec2173069089a87" candidate_uuid: "df994aaed44b4eda91d639dde8b5528f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" is_pre_election: true
I20260927 06:55:10.578238 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014d2a6cb6fa46299ec2173069089a87" candidate_uuid: "df994aaed44b4eda91d639dde8b5528f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb2661436744881b57bddbd30a1950a" is_pre_election: true
I20260927 06:55:10.578778 8154 raft_consensus.cc:2454] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df994aaed44b4eda91d639dde8b5528f in term 0.
I20260927 06:55:10.578778 8427 raft_consensus.cc:2454] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df994aaed44b4eda91d639dde8b5528f in term 0.
I20260927 06:55:10.579464 8221 leader_election.cc:302] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 366565fc5a9a4d3d806537f3bd3d9afe, deb2661436744881b57bddbd30a1950a, df994aaed44b4eda91d639dde8b5528f; no voters:
I20260927 06:55:10.579891 8509 raft_consensus.cc:2781] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:55:10.580041 8509 raft_consensus.cc:482] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:55:10.580139 8509 raft_consensus.cc:3037] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:10.582286 8509 raft_consensus.cc:504] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } }
I20260927 06:55:10.582826 8509 leader_election.cc:288] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [CANDIDATE]: Term 1 election: Requested vote from peers 366565fc5a9a4d3d806537f3bd3d9afe (127.0.97.1:34427), deb2661436744881b57bddbd30a1950a (127.0.97.3:39213)
I20260927 06:55:10.583225 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014d2a6cb6fa46299ec2173069089a87" candidate_uuid: "df994aaed44b4eda91d639dde8b5528f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "366565fc5a9a4d3d806537f3bd3d9afe"
I20260927 06:55:10.583302 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014d2a6cb6fa46299ec2173069089a87" candidate_uuid: "df994aaed44b4eda91d639dde8b5528f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb2661436744881b57bddbd30a1950a"
I20260927 06:55:10.583441 8154 raft_consensus.cc:3037] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:10.583528 8427 raft_consensus.cc:3037] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:10.585640 8154 raft_consensus.cc:2454] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df994aaed44b4eda91d639dde8b5528f in term 1.
I20260927 06:55:10.585860 8427 raft_consensus.cc:2454] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df994aaed44b4eda91d639dde8b5528f in term 1.
I20260927 06:55:10.586002 8222 leader_election.cc:302] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366565fc5a9a4d3d806537f3bd3d9afe, df994aaed44b4eda91d639dde8b5528f; no voters:
I20260927 06:55:10.586316 8509 raft_consensus.cc:2781] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:55:10.586783 8509 raft_consensus.cc:686] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [term 1 LEADER]: Becoming Leader. State: Replica: df994aaed44b4eda91d639dde8b5528f, State: Running, Role: LEADER
I20260927 06:55:10.587672 8509 consensus_queue.cc:237] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } }
I20260927 06:55:10.591645 7992 catalog_manager.cc:5495] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f reported cstate change: term changed from 0 to 1, leader changed from <none> to df994aaed44b4eda91d639dde8b5528f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "df994aaed44b4eda91d639dde8b5528f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } health_report { overall_health: HEALTHY } } }
I20260927 06:55:10.669672 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:10.664369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49214 (local address 127.0.97.2:35641)
0927 06:55:10.664610 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:10.664616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:10.664723 (+ 107us) server_negotiation.cc:407] Connection header received
0927 06:55:10.664978 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:10.664982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:10.665055 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:10.665200 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:55:10.665561 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.666334 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.667668 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:10.668061 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:10.668193 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:10.668622 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:55:10.669118 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:55:10.669275 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:10.669331 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 06:55:10.669405 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 06:55:10.677078 8427 raft_consensus.cc:1261] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Refusing update from remote peer df994aaed44b4eda91d639dde8b5528f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:55:10.677079 8154 raft_consensus.cc:1261] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 FOLLOWER]: Refusing update from remote peer df994aaed44b4eda91d639dde8b5528f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:55:10.677964 8512 consensus_queue.cc:1035] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [LEADER]: Connected to new peer: Peer: permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:55:10.678301 8509 consensus_queue.cc:1035] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [LEADER]: Connected to new peer: Peer: permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:55:10.705811 7992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:55:10.706087 7992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:55:10.709928 7992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a: Sending DeleteTablet for 3 replicas of tablet 014d2a6cb6fa46299ec2173069089a87
I20260927 06:55:10.711941 8129 tablet_service.cc:1514] Processing DeleteTablet for tablet 014d2a6cb6fa46299ec2173069089a87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:10 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44110
I20260927 06:55:10.712263 8407 tablet_service.cc:1514] Processing DeleteTablet for tablet 014d2a6cb6fa46299ec2173069089a87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:10 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50386
I20260927 06:55:10.713119 8268 tablet_service.cc:1514] Processing DeleteTablet for tablet 014d2a6cb6fa46299ec2173069089a87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:10 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49202
I20260927 06:55:10.713819 7991 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:55:10.714490 7991 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:55:10.719452 8517 mvcc.cc:205] Tried to move back new op lower bound from 7333855685324890112 to 7333855684970373120. Current Snapshot: MvccSnapshot[applied={T|T < 7333855685324890112}]
I20260927 06:55:10.725071 8267 tablet_service.cc:1467] Processing CreateTablet for tablet 682f396ec99f436189cc7de859ef6aa0 (DEFAULT_TABLE table=test-table [id=95e19ec8c0764bb0a4f08ad4ab0885ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:10.725818 8267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682f396ec99f436189cc7de859ef6aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:10.728202 8128 tablet_service.cc:1467] Processing CreateTablet for tablet 682f396ec99f436189cc7de859ef6aa0 (DEFAULT_TABLE table=test-table [id=95e19ec8c0764bb0a4f08ad4ab0885ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:10.728552 8406 tablet_service.cc:1467] Processing CreateTablet for tablet 682f396ec99f436189cc7de859ef6aa0 (DEFAULT_TABLE table=test-table [id=95e19ec8c0764bb0a4f08ad4ab0885ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:10.728861 8128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682f396ec99f436189cc7de859ef6aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:10.729045 8406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682f396ec99f436189cc7de859ef6aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:10.736833 8502 tablet_bootstrap.cc:492] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe: Bootstrap starting.
I20260927 06:55:10.738219 8525 tablet_replica.cc:330] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe: stopping tablet replica
I20260927 06:55:10.739038 8502 tablet_bootstrap.cc:654] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:10.748132 8504 tablet_bootstrap.cc:492] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f: Bootstrap starting.
I20260927 06:55:10.750303 8504 tablet_bootstrap.cc:654] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:10.754585 8501 tablet_bootstrap.cc:492] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a: Bootstrap starting.
I20260927 06:55:10.756548 8501 tablet_bootstrap.cc:654] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:10.762511 8501 tablet_bootstrap.cc:492] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a: No bootstrap required, opened a new log
I20260927 06:55:10.762688 8501 ts_tablet_manager.cc:1397] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260927 06:55:10.763691 8501 raft_consensus.cc:348] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:10.764027 8501 raft_consensus.cc:374] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:10.764135 8501 raft_consensus.cc:729] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deb2661436744881b57bddbd30a1950a, State: Initialized, Role: FOLLOWER
I20260927 06:55:10.764488 8501 consensus_queue.cc:260] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:10.767496 8526 tablet_replica.cc:330] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a: stopping tablet replica
I20260927 06:55:10.767997 8526 raft_consensus.cc:2229] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:55:10.768513 8526 raft_consensus.cc:2258] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:10.769395 8501 ts_tablet_manager.cc:1428] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260927 06:55:10.770121 8527 tablet_replica.cc:330] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f: stopping tablet replica
I20260927 06:55:10.770543 8527 raft_consensus.cc:2229] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:55:10.771191 8527 raft_consensus.cc:2258] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:10.772959 8502 tablet_bootstrap.cc:492] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe: No bootstrap required, opened a new log
I20260927 06:55:10.773296 8502 ts_tablet_manager.cc:1397] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.009s
I20260927 06:55:10.774281 8504 tablet_bootstrap.cc:492] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f: No bootstrap required, opened a new log
I20260927 06:55:10.774422 8504 ts_tablet_manager.cc:1397] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.000s
I20260927 06:55:10.774696 8502 raft_consensus.cc:348] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:10.774930 8502 raft_consensus.cc:374] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:10.775024 8502 raft_consensus.cc:729] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 366565fc5a9a4d3d806537f3bd3d9afe, State: Initialized, Role: FOLLOWER
I20260927 06:55:10.775254 8504 raft_consensus.cc:348] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:10.775269 8502 consensus_queue.cc:260] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:10.775440 8504 raft_consensus.cc:374] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:10.775544 8504 raft_consensus.cc:729] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df994aaed44b4eda91d639dde8b5528f, State: Initialized, Role: FOLLOWER
I20260927 06:55:10.775816 8502 ts_tablet_manager.cc:1428] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 06:55:10.775779 8504 consensus_queue.cc:260] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:10.776361 8504 ts_tablet_manager.cc:1428] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 06:55:10.778328 8525 raft_consensus.cc:2229] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:55:10.781243 8526 ts_tablet_manager.cc:1905] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:10.781548 8525 raft_consensus.cc:2258] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:10.783339 8525 ts_tablet_manager.cc:1905] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:10.780339 8527 ts_tablet_manager.cc:1905] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:10.786396 8526 ts_tablet_manager.cc:1918] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:55:10.786537 8526 log.cc:1196] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/014d2a6cb6fa46299ec2173069089a87
I20260927 06:55:10.786948 8526 ts_tablet_manager.cc:1939] T 014d2a6cb6fa46299ec2173069089a87 P deb2661436744881b57bddbd30a1950a: Deleting consensus metadata
I20260927 06:55:10.788264 7978 catalog_manager.cc:4841] TS deb2661436744881b57bddbd30a1950a (127.0.97.3:39213): tablet 014d2a6cb6fa46299ec2173069089a87 (table test-table [id=ce438088505c47fa9c1f0ba0e2dcb5fb]) successfully deleted
I20260927 06:55:10.793023 8525 ts_tablet_manager.cc:1918] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:55:10.793180 8525 log.cc:1196] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/014d2a6cb6fa46299ec2173069089a87
I20260927 06:55:10.793188 8527 ts_tablet_manager.cc:1918] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:55:10.793349 8527 log.cc:1196] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/014d2a6cb6fa46299ec2173069089a87
I20260927 06:55:10.793646 8525 ts_tablet_manager.cc:1939] T 014d2a6cb6fa46299ec2173069089a87 P 366565fc5a9a4d3d806537f3bd3d9afe: Deleting consensus metadata
I20260927 06:55:10.793905 8527 ts_tablet_manager.cc:1939] T 014d2a6cb6fa46299ec2173069089a87 P df994aaed44b4eda91d639dde8b5528f: Deleting consensus metadata
I20260927 06:55:10.795058 7979 catalog_manager.cc:4841] TS 366565fc5a9a4d3d806537f3bd3d9afe (127.0.97.1:34427): tablet 014d2a6cb6fa46299ec2173069089a87 (table test-table [id=ce438088505c47fa9c1f0ba0e2dcb5fb]) successfully deleted
I20260927 06:55:10.795356 7976 catalog_manager.cc:4841] TS df994aaed44b4eda91d639dde8b5528f (127.0.97.2:35641): tablet 014d2a6cb6fa46299ec2173069089a87 (table test-table [id=ce438088505c47fa9c1f0ba0e2dcb5fb]) successfully deleted
I20260927 06:55:11.055794 8512 raft_consensus.cc:482] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:55:11.056097 8512 raft_consensus.cc:504] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.056813 8512 leader_election.cc:288] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 366565fc5a9a4d3d806537f3bd3d9afe (127.0.97.1:34427), deb2661436744881b57bddbd30a1950a (127.0.97.3:39213)
I20260927 06:55:11.057476 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682f396ec99f436189cc7de859ef6aa0" candidate_uuid: "df994aaed44b4eda91d639dde8b5528f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb2661436744881b57bddbd30a1950a" is_pre_election: true
I20260927 06:55:11.057538 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682f396ec99f436189cc7de859ef6aa0" candidate_uuid: "df994aaed44b4eda91d639dde8b5528f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" is_pre_election: true
I20260927 06:55:11.057780 8154 raft_consensus.cc:2454] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df994aaed44b4eda91d639dde8b5528f in term 0.
I20260927 06:55:11.057771 8427 raft_consensus.cc:2454] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df994aaed44b4eda91d639dde8b5528f in term 0.
I20260927 06:55:11.058182 8222 leader_election.cc:302] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366565fc5a9a4d3d806537f3bd3d9afe, df994aaed44b4eda91d639dde8b5528f; no voters:
I20260927 06:55:11.058462 8512 raft_consensus.cc:2781] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:55:11.058588 8512 raft_consensus.cc:482] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:55:11.058706 8512 raft_consensus.cc:3037] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:11.060883 8512 raft_consensus.cc:504] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.061435 8512 leader_election.cc:288] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [CANDIDATE]: Term 1 election: Requested vote from peers 366565fc5a9a4d3d806537f3bd3d9afe (127.0.97.1:34427), deb2661436744881b57bddbd30a1950a (127.0.97.3:39213)
I20260927 06:55:11.061828 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682f396ec99f436189cc7de859ef6aa0" candidate_uuid: "df994aaed44b4eda91d639dde8b5528f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "366565fc5a9a4d3d806537f3bd3d9afe"
I20260927 06:55:11.061888 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682f396ec99f436189cc7de859ef6aa0" candidate_uuid: "df994aaed44b4eda91d639dde8b5528f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb2661436744881b57bddbd30a1950a"
I20260927 06:55:11.062048 8154 raft_consensus.cc:3037] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:11.062090 8427 raft_consensus.cc:3037] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:11.063928 8427 raft_consensus.cc:2454] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df994aaed44b4eda91d639dde8b5528f in term 1.
I20260927 06:55:11.064273 8221 leader_election.cc:302] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: deb2661436744881b57bddbd30a1950a, df994aaed44b4eda91d639dde8b5528f; no voters:
I20260927 06:55:11.064508 8512 raft_consensus.cc:2781] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:55:11.064682 8512 raft_consensus.cc:686] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [term 1 LEADER]: Becoming Leader. State: Replica: df994aaed44b4eda91d639dde8b5528f, State: Running, Role: LEADER
I20260927 06:55:11.064972 8154 raft_consensus.cc:2454] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df994aaed44b4eda91d639dde8b5528f in term 1.
I20260927 06:55:11.065223 8512 consensus_queue.cc:237] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.067999 7992 catalog_manager.cc:5495] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f reported cstate change: term changed from 0 to 1, leader changed from <none> to df994aaed44b4eda91d639dde8b5528f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "df994aaed44b4eda91d639dde8b5528f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } health_report { overall_health: UNKNOWN } } }
Sep 27 06:55:11 dist-test-slave-f892 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492108, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 06:55:11.132143 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:11.116187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41405 (local address 127.0.0.1:33536)
0927 06:55:11.116891 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:11.116912 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:11.117066 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:11.118765 (+ 1699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:11.118769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:11.119408 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:11.119597 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:11.119606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.121228 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.121231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:11.122213 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:11.122224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.123099 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.123103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:11.123261 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.123932 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:11.123948 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:11.126207 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:11.129881 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:11.129890 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:11.129895 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:11.130331 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:11.130685 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:11.130688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:11.130690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:11.130813 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:11.131416 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:11.131422 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:11.131548 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 06:55:11.131677 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 06:55:11.131867 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":29,"thread_start_us":370,"threads_started":1}
I20260927 06:55:11.132470 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:11.116230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.0.97.62:41405)
0927 06:55:11.118452 (+ 2222us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:11.118459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:11.118480 (+ 21us) server_negotiation.cc:407] Connection header received
0927 06:55:11.118532 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:11.118537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:11.118605 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:11.118721 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:11.119771 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.121089 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.122340 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.122960 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.123087 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.126381 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:11.126388 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:11.126393 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:11.126426 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:11.129751 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:11.130440 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:11.130445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:11.130449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:11.130590 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:11.130956 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:11.130964 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:11.130968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:11.131299 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:11.131439 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:11.131814 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 06:55:11.132015 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":2095,"thread_start_us":1948,"threads_started":1}
I20260927 06:55:11.137038 7992 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33536:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 06:55:11.137988 7992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:55:11.144223 8129 tablet_service.cc:1467] Processing CreateTablet for tablet 2c7d080bd0e64e64969e5b1a2ab71975 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5f442378e2ca4144bf900ee0c8774fd3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 06:55:11.144424 8268 tablet_service.cc:1467] Processing CreateTablet for tablet 2c7d080bd0e64e64969e5b1a2ab71975 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5f442378e2ca4144bf900ee0c8774fd3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 06:55:11.144739 8129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c7d080bd0e64e64969e5b1a2ab71975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:11.144970 8268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c7d080bd0e64e64969e5b1a2ab71975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:11.148145 8407 tablet_service.cc:1467] Processing CreateTablet for tablet 2c7d080bd0e64e64969e5b1a2ab71975 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5f442378e2ca4144bf900ee0c8774fd3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 06:55:11.148646 8407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c7d080bd0e64e64969e5b1a2ab71975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:11.151346 8504 tablet_bootstrap.cc:492] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f: Bootstrap starting.
I20260927 06:55:11.151499 8502 tablet_bootstrap.cc:492] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe: Bootstrap starting.
I20260927 06:55:11.153002 8504 tablet_bootstrap.cc:654] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:11.153048 8502 tablet_bootstrap.cc:654] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:11.154529 8501 tablet_bootstrap.cc:492] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a: Bootstrap starting.
I20260927 06:55:11.155189 8502 tablet_bootstrap.cc:492] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe: No bootstrap required, opened a new log
I20260927 06:55:11.155344 8502 ts_tablet_manager.cc:1397] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 06:55:11.155717 8504 tablet_bootstrap.cc:492] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f: No bootstrap required, opened a new log
I20260927 06:55:11.155861 8504 ts_tablet_manager.cc:1397] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 06:55:11.156253 8501 tablet_bootstrap.cc:654] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:11.156167 8502 raft_consensus.cc:348] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.156391 8502 raft_consensus.cc:374] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:11.156476 8502 raft_consensus.cc:729] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 366565fc5a9a4d3d806537f3bd3d9afe, State: Initialized, Role: FOLLOWER
I20260927 06:55:11.156555 8504 raft_consensus.cc:348] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.156777 8504 raft_consensus.cc:374] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:11.156852 8504 raft_consensus.cc:729] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df994aaed44b4eda91d639dde8b5528f, State: Initialized, Role: FOLLOWER
I20260927 06:55:11.156764 8502 consensus_queue.cc:260] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.157079 8504 consensus_queue.cc:260] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.157382 8502 ts_tablet_manager.cc:1428] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 06:55:11.157326 8508 tablet_replica.cc:439] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } }
I20260927 06:55:11.157598 8504 ts_tablet_manager.cc:1428] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 06:55:11.157675 8508 tablet_replica.cc:442] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:55:11.158620 8512 tablet_replica.cc:439] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } }
I20260927 06:55:11.158845 8512 tablet_replica.cc:442] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:55:11.163019 8501 tablet_bootstrap.cc:492] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a: No bootstrap required, opened a new log
I20260927 06:55:11.163193 8501 ts_tablet_manager.cc:1397] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 06:55:11.163743 8501 raft_consensus.cc:348] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.163944 8501 raft_consensus.cc:374] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:11.164036 8501 raft_consensus.cc:729] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deb2661436744881b57bddbd30a1950a, State: Initialized, Role: FOLLOWER
I20260927 06:55:11.164259 8501 consensus_queue.cc:260] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.164777 8501 ts_tablet_manager.cc:1428] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 06:55:11.164678 8507 tablet_replica.cc:439] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } }
I20260927 06:55:11.165045 8507 tablet_replica.cc:442] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:55:11.173062 8512 raft_consensus.cc:482] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:55:11.173208 8512 raft_consensus.cc:504] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.173744 8512 leader_election.cc:288] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 366565fc5a9a4d3d806537f3bd3d9afe (127.0.97.1:34427), deb2661436744881b57bddbd30a1950a (127.0.97.3:39213)
I20260927 06:55:11.174139 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c7d080bd0e64e64969e5b1a2ab71975" candidate_uuid: "df994aaed44b4eda91d639dde8b5528f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" is_pre_election: true
I20260927 06:55:11.174208 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c7d080bd0e64e64969e5b1a2ab71975" candidate_uuid: "df994aaed44b4eda91d639dde8b5528f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb2661436744881b57bddbd30a1950a" is_pre_election: true
I20260927 06:55:11.174391 8154 raft_consensus.cc:2454] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df994aaed44b4eda91d639dde8b5528f in term 0.
I20260927 06:55:11.174474 8427 raft_consensus.cc:2454] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df994aaed44b4eda91d639dde8b5528f in term 0.
I20260927 06:55:11.174808 8222 leader_election.cc:302] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366565fc5a9a4d3d806537f3bd3d9afe, df994aaed44b4eda91d639dde8b5528f; no voters:
I20260927 06:55:11.175038 8512 raft_consensus.cc:2781] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:55:11.175146 8512 raft_consensus.cc:482] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:55:11.175222 8512 raft_consensus.cc:3037] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:11.177242 8512 raft_consensus.cc:504] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.177784 8512 leader_election.cc:288] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [CANDIDATE]: Term 1 election: Requested vote from peers 366565fc5a9a4d3d806537f3bd3d9afe (127.0.97.1:34427), deb2661436744881b57bddbd30a1950a (127.0.97.3:39213)
I20260927 06:55:11.178160 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c7d080bd0e64e64969e5b1a2ab71975" candidate_uuid: "df994aaed44b4eda91d639dde8b5528f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "366565fc5a9a4d3d806537f3bd3d9afe"
I20260927 06:55:11.178242 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c7d080bd0e64e64969e5b1a2ab71975" candidate_uuid: "df994aaed44b4eda91d639dde8b5528f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb2661436744881b57bddbd30a1950a"
I20260927 06:55:11.178378 8154 raft_consensus.cc:3037] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:11.178448 8427 raft_consensus.cc:3037] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:11.181152 8154 raft_consensus.cc:2454] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df994aaed44b4eda91d639dde8b5528f in term 1.
I20260927 06:55:11.181152 8427 raft_consensus.cc:2454] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df994aaed44b4eda91d639dde8b5528f in term 1.
I20260927 06:55:11.181533 8221 leader_election.cc:302] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 366565fc5a9a4d3d806537f3bd3d9afe, deb2661436744881b57bddbd30a1950a, df994aaed44b4eda91d639dde8b5528f; no voters:
I20260927 06:55:11.181792 8512 raft_consensus.cc:2781] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:55:11.181936 8512 raft_consensus.cc:686] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [term 1 LEADER]: Becoming Leader. State: Replica: df994aaed44b4eda91d639dde8b5528f, State: Running, Role: LEADER
I20260927 06:55:11.182214 8512 consensus_queue.cc:237] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.183876 8520 tablet_replica.cc:439] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f: TxnStatusTablet state changed. Reason: New leader df994aaed44b4eda91d639dde8b5528f. Latest consensus state: current_term: 1 leader_uuid: "df994aaed44b4eda91d639dde8b5528f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } }
I20260927 06:55:11.184165 8520 tablet_replica.cc:442] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f: This TxnStatusTablet replica's current role is: LEADER
I20260927 06:55:11.184772 8543 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 06:55:11.185009 8543 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 06:55:11.185278 7992 catalog_manager.cc:5495] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f reported cstate change: term changed from 0 to 1, leader changed from <none> to df994aaed44b4eda91d639dde8b5528f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "df994aaed44b4eda91d639dde8b5528f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } health_report { overall_health: UNKNOWN } } }
I20260927 06:55:11.217368 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:11.198720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35641 (local address 127.0.0.1:49216)
0927 06:55:11.199122 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:11.199142 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:11.199264 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:11.212194 (+ 12930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:11.212200 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:11.212216 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 06:55:11.212411 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:11.212422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.213717 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.213724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:11.215335 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:11.215350 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.216105 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.216111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:11.216310 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.216388 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 06:55:11.216837 (+ 449us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 06:55:11.216866 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 06:55:11.216919 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 06:55:11.217065 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243}
I20260927 06:55:11.218171 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:11.199516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49216 (local address 127.0.97.2:35641)
0927 06:55:11.211837 (+ 12321us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:11.211842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:11.211859 (+ 17us) server_negotiation.cc:407] Connection header received
0927 06:55:11.211913 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:11.211916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:11.211972 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:11.212152 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:55:11.212645 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.213582 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.215558 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.215954 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.216093 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.216580 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:55:11.216725 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:55:11.217697 (+ 972us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:11.217746 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 06:55:11.217814 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12220,"thread_start_us":12097,"threads_started":1}
I20260927 06:55:11.228174 8427 raft_consensus.cc:1261] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Refusing update from remote peer df994aaed44b4eda91d639dde8b5528f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:55:11.228785 8520 consensus_queue.cc:1035] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [LEADER]: Connected to new peer: Peer: permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:55:11.229224 8154 raft_consensus.cc:1261] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 FOLLOWER]: Refusing update from remote peer df994aaed44b4eda91d639dde8b5528f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:55:11.229857 8520 consensus_queue.cc:1035] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f [LEADER]: Connected to new peer: Peer: permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:55:11.238700 8507 tablet_replica.cc:439] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a: TxnStatusTablet state changed. Reason: New leader df994aaed44b4eda91d639dde8b5528f. Latest consensus state: current_term: 1 leader_uuid: "df994aaed44b4eda91d639dde8b5528f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } }
I20260927 06:55:11.238978 8507 tablet_replica.cc:442] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:55:11.243400 8546 mvcc.cc:205] Tried to move back new op lower bound from 7333855687584825344 to 7333855687404756992. Current Snapshot: MvccSnapshot[applied={T|T < 7333855687584825344}]
I20260927 06:55:11.244254 8520 tablet_replica.cc:439] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df994aaed44b4eda91d639dde8b5528f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } }
I20260927 06:55:11.244540 8520 tablet_replica.cc:442] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f: This TxnStatusTablet replica's current role is: LEADER
I20260927 06:55:11.252895 8508 tablet_replica.cc:439] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe: TxnStatusTablet state changed. Reason: New leader df994aaed44b4eda91d639dde8b5528f. Latest consensus state: current_term: 1 leader_uuid: "df994aaed44b4eda91d639dde8b5528f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } }
I20260927 06:55:11.252815 8512 tablet_replica.cc:439] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df994aaed44b4eda91d639dde8b5528f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } }
I20260927 06:55:11.253149 8508 tablet_replica.cc:442] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:55:11.253180 8512 tablet_replica.cc:442] T 2c7d080bd0e64e64969e5b1a2ab71975 P df994aaed44b4eda91d639dde8b5528f: This TxnStatusTablet replica's current role is: LEADER
I20260927 06:55:11.257344 8508 tablet_replica.cc:439] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df994aaed44b4eda91d639dde8b5528f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } }
I20260927 06:55:11.257581 8508 tablet_replica.cc:442] T 2c7d080bd0e64e64969e5b1a2ab71975 P 366565fc5a9a4d3d806537f3bd3d9afe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:55:11.257644 8507 tablet_replica.cc:439] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df994aaed44b4eda91d639dde8b5528f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } } }
I20260927 06:55:11.257859 8507 tablet_replica.cc:442] T 2c7d080bd0e64e64969e5b1a2ab71975 P deb2661436744881b57bddbd30a1950a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:55:11.270744 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:11.265062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36977 (local address 127.0.97.2:35641)
0927 06:55:11.265273 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:11.265280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:11.265743 (+ 463us) server_negotiation.cc:407] Connection header received
0927 06:55:11.265964 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:11.265969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:11.266093 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:11.266179 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:55:11.266735 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.267568 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.269021 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.269446 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.269576 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.269961 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:55:11.270127 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:55:11.270219 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:11.270341 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 06:55:11.270407 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":76}
I20260927 06:55:11.270838 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:11.264812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35641 (local address 127.0.97.2:36977)
0927 06:55:11.265647 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:11.265661 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:11.265840 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:11.266237 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:11.266244 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:11.266255 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 06:55:11.266532 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:11.266539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.267682 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.267687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:11.268886 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:11.268900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.269620 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.269629 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:11.269749 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.269796 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 06:55:11.270217 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 06:55:11.270227 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 06:55:11.270315 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 06:55:11.270432 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":401,"threads_started":1}
I20260927 06:55:11.279490 8427 raft_consensus.cc:1261] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Refusing update from remote peer df994aaed44b4eda91d639dde8b5528f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:55:11.279954 8153 raft_consensus.cc:1261] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 FOLLOWER]: Refusing update from remote peer df994aaed44b4eda91d639dde8b5528f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:55:11.280261 8556 consensus_queue.cc:1035] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [LEADER]: Connected to new peer: Peer: permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:55:11.280686 8512 consensus_queue.cc:1035] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [LEADER]: Connected to new peer: Peer: permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:55:11.287133 8547 mvcc.cc:205] Tried to move back new op lower bound from 7333855687796666368 to 7333855686925570048. Current Snapshot: MvccSnapshot[applied={T|T < 7333855687796666368}]
I20260927 06:55:11.337064 7992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:55:11.337275 7992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:55:11.339867 7992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a: Sending DeleteTablet for 3 replicas of tablet 682f396ec99f436189cc7de859ef6aa0
I20260927 06:55:11.340536 8129 tablet_service.cc:1514] Processing DeleteTablet for tablet 682f396ec99f436189cc7de859ef6aa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44110
I20260927 06:55:11.340731 8407 tablet_service.cc:1514] Processing DeleteTablet for tablet 682f396ec99f436189cc7de859ef6aa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50386
I20260927 06:55:11.341143 8560 tablet_replica.cc:330] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe: stopping tablet replica
I20260927 06:55:11.341387 8560 raft_consensus.cc:2229] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:55:11.341550 8267 tablet_service.cc:1514] Processing DeleteTablet for tablet 682f396ec99f436189cc7de859ef6aa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49202
I20260927 06:55:11.341610 8560 raft_consensus.cc:2258] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:11.342742 8560 ts_tablet_manager.cc:1905] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:11.352095 8560 ts_tablet_manager.cc:1918] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:55:11.352258 8560 log.cc:1196] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/682f396ec99f436189cc7de859ef6aa0
I20260927 06:55:11.352824 8560 ts_tablet_manager.cc:1939] T 682f396ec99f436189cc7de859ef6aa0 P 366565fc5a9a4d3d806537f3bd3d9afe: Deleting consensus metadata
I20260927 06:55:11.353643 8561 tablet_replica.cc:330] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a: stopping tablet replica
I20260927 06:55:11.353963 8561 raft_consensus.cc:2229] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:55:11.353928 7979 catalog_manager.cc:4841] TS 366565fc5a9a4d3d806537f3bd3d9afe (127.0.97.1:34427): tablet 682f396ec99f436189cc7de859ef6aa0 (table test-table [id=95e19ec8c0764bb0a4f08ad4ab0885ad]) successfully deleted
I20260927 06:55:11.354274 8561 raft_consensus.cc:2258] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:11.355127 8561 ts_tablet_manager.cc:1905] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:11.361310 8561 ts_tablet_manager.cc:1918] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:55:11.361451 8561 log.cc:1196] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/682f396ec99f436189cc7de859ef6aa0
I20260927 06:55:11.361857 8561 ts_tablet_manager.cc:1939] T 682f396ec99f436189cc7de859ef6aa0 P deb2661436744881b57bddbd30a1950a: Deleting consensus metadata
I20260927 06:55:11.363176 7978 catalog_manager.cc:4841] TS deb2661436744881b57bddbd30a1950a (127.0.97.3:39213): tablet 682f396ec99f436189cc7de859ef6aa0 (table test-table [id=95e19ec8c0764bb0a4f08ad4ab0885ad]) successfully deleted
I20260927 06:55:11.364259 8562 tablet_replica.cc:330] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f: stopping tablet replica
I20260927 06:55:11.364478 8562 raft_consensus.cc:2229] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:55:11.364959 8562 raft_consensus.cc:2258] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:11.365856 8562 ts_tablet_manager.cc:1905] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:11.371022 8562 ts_tablet_manager.cc:1918] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:55:11.371160 8562 log.cc:1196] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/682f396ec99f436189cc7de859ef6aa0
I20260927 06:55:11.371668 8562 ts_tablet_manager.cc:1939] T 682f396ec99f436189cc7de859ef6aa0 P df994aaed44b4eda91d639dde8b5528f: Deleting consensus metadata
I20260927 06:55:11.372560 7976 catalog_manager.cc:4841] TS df994aaed44b4eda91d639dde8b5528f (127.0.97.2:35641): tablet 682f396ec99f436189cc7de859ef6aa0 (table test-table [id=95e19ec8c0764bb0a4f08ad4ab0885ad]) successfully deleted
Sep 27 06:55:11 dist-test-slave-f892 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 06:55:11 dist-test-slave-f892 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:11 dist-test-slave-f892 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 06:55:11.406666 8538 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:11.396655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33542 (local address 127.0.97.62:41405)
0927 06:55:11.396854 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:11.396860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:11.397313 (+ 453us) server_negotiation.cc:407] Connection header received
0927 06:55:11.397455 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:11.397462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:11.397544 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:11.397671 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:11.398935 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.400075 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.401290 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.401682 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.401808 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.406421 (+ 4613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33542: BlockingRecv error: recv got EOF from 127.0.0.1:33542 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 06:55:11 dist-test-slave-f892 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 06:55:11.423480 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:11.409328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.0.97.62:41405)
0927 06:55:11.409499 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:11.409505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:11.409766 (+ 261us) server_negotiation.cc:407] Connection header received
0927 06:55:11.409899 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:11.409905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:11.409976 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:11.410107 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:11.411113 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.412245 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.413581 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.414273 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.414390 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.417833 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:11.417840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:11.417845 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:11.417881 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:11.421062 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:11.421601 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:11.421606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:11.421610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:11.421756 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:11.422160 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:11.422165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:11.422168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:11.422490 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:11.422723 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:11.422892 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 06:55:11.423098 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 06:55:11.426692 7992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:55:11.427278 7992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:55:11.433380 8129 tablet_service.cc:1467] Processing CreateTablet for tablet 13ad6c06e33b4ac88a96f40c7166a076 (DEFAULT_TABLE table=test-table [id=9d9f312a5001456dbefd2c6a0bf8f9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:11.433434 8267 tablet_service.cc:1467] Processing CreateTablet for tablet 13ad6c06e33b4ac88a96f40c7166a076 (DEFAULT_TABLE table=test-table [id=9d9f312a5001456dbefd2c6a0bf8f9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:11.433713 8407 tablet_service.cc:1467] Processing CreateTablet for tablet 13ad6c06e33b4ac88a96f40c7166a076 (DEFAULT_TABLE table=test-table [id=9d9f312a5001456dbefd2c6a0bf8f9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:11.433837 8129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ad6c06e33b4ac88a96f40c7166a076. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:11.433898 8267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ad6c06e33b4ac88a96f40c7166a076. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:11.434162 8407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ad6c06e33b4ac88a96f40c7166a076. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:11.440371 8501 tablet_bootstrap.cc:492] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a: Bootstrap starting.
I20260927 06:55:11.440455 8502 tablet_bootstrap.cc:492] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe: Bootstrap starting.
I20260927 06:55:11.441896 8501 tablet_bootstrap.cc:654] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:11.441949 8502 tablet_bootstrap.cc:654] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:11.442884 8504 tablet_bootstrap.cc:492] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f: Bootstrap starting.
I20260927 06:55:11.444347 8504 tablet_bootstrap.cc:654] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:11.446275 8501 tablet_bootstrap.cc:492] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a: No bootstrap required, opened a new log
I20260927 06:55:11.446413 8501 ts_tablet_manager.cc:1397] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 06:55:11.447242 8504 tablet_bootstrap.cc:492] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f: No bootstrap required, opened a new log
I20260927 06:55:11.447152 8501 raft_consensus.cc:348] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.447376 8504 ts_tablet_manager.cc:1397] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 06:55:11.447441 8501 raft_consensus.cc:374] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:11.447527 8501 raft_consensus.cc:729] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deb2661436744881b57bddbd30a1950a, State: Initialized, Role: FOLLOWER
I20260927 06:55:11.447729 8501 consensus_queue.cc:260] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.448060 8504 raft_consensus.cc:348] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.448303 8504 raft_consensus.cc:374] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:11.448395 8504 raft_consensus.cc:729] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df994aaed44b4eda91d639dde8b5528f, State: Initialized, Role: FOLLOWER
I20260927 06:55:11.448616 8504 consensus_queue.cc:260] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.449627 8504 ts_tablet_manager.cc:1428] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 06:55:11.452459 8501 ts_tablet_manager.cc:1428] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 06:55:11.454373 8502 tablet_bootstrap.cc:492] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe: No bootstrap required, opened a new log
I20260927 06:55:11.454510 8502 ts_tablet_manager.cc:1397] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20260927 06:55:11.455204 8502 raft_consensus.cc:348] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.455462 8502 raft_consensus.cc:374] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:11.455549 8502 raft_consensus.cc:729] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 366565fc5a9a4d3d806537f3bd3d9afe, State: Initialized, Role: FOLLOWER
I20260927 06:55:11.455746 8502 consensus_queue.cc:260] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.456274 8502 ts_tablet_manager.cc:1428] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 06:55:11.458086 8508 raft_consensus.cc:482] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:55:11.458276 8508 raft_consensus.cc:504] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.460019 8508 leader_election.cc:288] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df994aaed44b4eda91d639dde8b5528f (127.0.97.2:35641), deb2661436744881b57bddbd30a1950a (127.0.97.3:39213)
I20260927 06:55:11.469081 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:11.460675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57729 (local address 127.0.97.2:35641)
0927 06:55:11.460846 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:11.460851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:11.463580 (+ 2729us) server_negotiation.cc:407] Connection header received
0927 06:55:11.463753 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:11.463758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:11.463815 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:11.463916 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:11.464462 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.465359 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.467208 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.468180 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.468331 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.468435 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:11.468671 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 06:55:11.468791 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 06:55:11.469098 8579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:11.460262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35641 (local address 127.0.97.1:57729)
0927 06:55:11.463477 (+ 3215us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:11.463492 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:11.463641 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:11.463928 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:11.463932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:11.463952 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:11.464292 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:11.464303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.465488 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.465491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:11.467010 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:11.467021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.468361 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.468367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:11.468526 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.468548 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 06:55:11.468651 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 06:55:11.468776 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3110,"thread_start_us":2947,"threads_started":1}
I20260927 06:55:11.469813 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:11.460236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39213 (local address 127.0.97.1:51291)
0927 06:55:11.460712 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:11.460730 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:11.460849 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:11.462847 (+ 1998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:11.462852 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:11.462880 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:11.463293 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:11.463306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.464803 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.464808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:11.466907 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:11.466921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.468636 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.468642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:11.468813 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.468845 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 06:55:11.468935 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 06:55:11.469062 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":1,"thread_start_us":399,"threads_started":1}
I20260927 06:55:11.469909 8293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ad6c06e33b4ac88a96f40c7166a076" candidate_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df994aaed44b4eda91d639dde8b5528f" is_pre_election: true
I20260927 06:55:11.470371 8293 raft_consensus.cc:2454] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 366565fc5a9a4d3d806537f3bd3d9afe in term 0.
I20260927 06:55:11.470528 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:11.460542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51291 (local address 127.0.97.3:39213)
0927 06:55:11.462482 (+ 1940us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:11.462491 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:11.462519 (+ 28us) server_negotiation.cc:407] Connection header received
0927 06:55:11.462588 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:11.462594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:11.462680 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:11.462811 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:11.463525 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.464646 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.467080 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.468265 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.468379 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.468496 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:11.470120 (+ 1624us) server_negotiation.cc:299] Negotiation successful
0927 06:55:11.470239 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1794,"thread_start_us":1656,"threads_started":1}
I20260927 06:55:11.470892 8079 leader_election.cc:302] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366565fc5a9a4d3d806537f3bd3d9afe, df994aaed44b4eda91d639dde8b5528f; no voters:
I20260927 06:55:11.470948 8426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ad6c06e33b4ac88a96f40c7166a076" candidate_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb2661436744881b57bddbd30a1950a" is_pre_election: true
I20260927 06:55:11.471196 8426 raft_consensus.cc:2454] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 366565fc5a9a4d3d806537f3bd3d9afe in term 0.
I20260927 06:55:11.471381 8508 raft_consensus.cc:2781] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:55:11.471504 8508 raft_consensus.cc:482] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:55:11.471591 8508 raft_consensus.cc:3037] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:11.473713 8508 raft_consensus.cc:504] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.474253 8508 leader_election.cc:288] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [CANDIDATE]: Term 1 election: Requested vote from peers df994aaed44b4eda91d639dde8b5528f (127.0.97.2:35641), deb2661436744881b57bddbd30a1950a (127.0.97.3:39213)
I20260927 06:55:11.474767 8426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ad6c06e33b4ac88a96f40c7166a076" candidate_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb2661436744881b57bddbd30a1950a"
I20260927 06:55:11.474920 8426 raft_consensus.cc:3037] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:11.476725 8426 raft_consensus.cc:2454] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 366565fc5a9a4d3d806537f3bd3d9afe in term 1.
I20260927 06:55:11.477077 8081 leader_election.cc:302] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366565fc5a9a4d3d806537f3bd3d9afe, deb2661436744881b57bddbd30a1950a; no voters:
I20260927 06:55:11.477087 8293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ad6c06e33b4ac88a96f40c7166a076" candidate_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df994aaed44b4eda91d639dde8b5528f"
I20260927 06:55:11.477288 8293 raft_consensus.cc:3037] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:11.477349 8508 raft_consensus.cc:2781] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:55:11.477771 8508 raft_consensus.cc:686] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 LEADER]: Becoming Leader. State: Replica: 366565fc5a9a4d3d806537f3bd3d9afe, State: Running, Role: LEADER
I20260927 06:55:11.478163 8508 consensus_queue.cc:237] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.480823 8293 raft_consensus.cc:2454] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 366565fc5a9a4d3d806537f3bd3d9afe in term 1.
I20260927 06:55:11.483134 7991 catalog_manager.cc:5495] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe reported cstate change: term changed from 0 to 1, leader changed from <none> to 366565fc5a9a4d3d806537f3bd3d9afe (127.0.97.1). New cstate: current_term: 1 leader_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } health_report { overall_health: UNKNOWN } } }
I20260927 06:55:11.500908 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:11.495352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44118 (local address 127.0.97.1:34427)
0927 06:55:11.495892 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:11.495896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:11.495911 (+ 15us) server_negotiation.cc:407] Connection header received
0927 06:55:11.495977 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:11.495980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:11.496042 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:11.496202 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:55:11.496622 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.497418 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.498824 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.499224 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.499352 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.499754 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:55:11.500341 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:55:11.500480 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:11.500550 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 06:55:11.500620 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":317,"threads_started":1}
I20260927 06:55:11.504290 8426 raft_consensus.cc:1261] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Refusing update from remote peer 366565fc5a9a4d3d806537f3bd3d9afe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:55:11.504572 8293 raft_consensus.cc:1261] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f [term 1 FOLLOWER]: Refusing update from remote peer 366565fc5a9a4d3d806537f3bd3d9afe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:55:11.504974 8581 consensus_queue.cc:1035] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [LEADER]: Connected to new peer: Peer: permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:55:11.505355 8508 consensus_queue.cc:1035] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [LEADER]: Connected to new peer: Peer: permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:55:11.531881 7992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:55:11.532161 7992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:55:11.535606 7992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a: Sending DeleteTablet for 3 replicas of tablet 13ad6c06e33b4ac88a96f40c7166a076
I20260927 06:55:11.536363 8129 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ad6c06e33b4ac88a96f40c7166a076 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44110
I20260927 06:55:11.536520 8407 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ad6c06e33b4ac88a96f40c7166a076 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50386
I20260927 06:55:11.536662 8560 tablet_replica.cc:330] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe: stopping tablet replica
I20260927 06:55:11.536785 8561 tablet_replica.cc:330] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a: stopping tablet replica
I20260927 06:55:11.536865 8560 raft_consensus.cc:2229] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:55:11.536967 8561 raft_consensus.cc:2229] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:55:11.537166 8561 raft_consensus.cc:2258] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:11.537336 8560 raft_consensus.cc:2258] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:11.538262 8561 ts_tablet_manager.cc:1905] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:11.538352 8267 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ad6c06e33b4ac88a96f40c7166a076 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49202
I20260927 06:55:11.538652 8562 tablet_replica.cc:330] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f: stopping tablet replica
I20260927 06:55:11.538853 8562 raft_consensus.cc:2229] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:55:11.539083 8562 raft_consensus.cc:2258] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:11.540366 8560 ts_tablet_manager.cc:1905] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:11.543560 7991 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:55:11.544209 7991 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:55:11.549170 8561 ts_tablet_manager.cc:1918] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:55:11.549320 8561 log.cc:1196] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/13ad6c06e33b4ac88a96f40c7166a076
I20260927 06:55:11.549778 8561 ts_tablet_manager.cc:1939] T 13ad6c06e33b4ac88a96f40c7166a076 P deb2661436744881b57bddbd30a1950a: Deleting consensus metadata
I20260927 06:55:11.550863 7978 catalog_manager.cc:4841] TS deb2661436744881b57bddbd30a1950a (127.0.97.3:39213): tablet 13ad6c06e33b4ac88a96f40c7166a076 (table test-table [id=9d9f312a5001456dbefd2c6a0bf8f9ff]) successfully deleted
I20260927 06:55:11.552922 8407 tablet_service.cc:1467] Processing CreateTablet for tablet 441575de489a4660bce5a77bd0e73bbb (DEFAULT_TABLE table=test-table [id=59d7a6fe1ea443d08973bb8baf5797e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:11.553311 8407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441575de489a4660bce5a77bd0e73bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:11.554811 8129 tablet_service.cc:1467] Processing CreateTablet for tablet 441575de489a4660bce5a77bd0e73bbb (DEFAULT_TABLE table=test-table [id=59d7a6fe1ea443d08973bb8baf5797e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:11.555208 8129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441575de489a4660bce5a77bd0e73bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:11.558976 8267 tablet_service.cc:1467] Processing CreateTablet for tablet 441575de489a4660bce5a77bd0e73bbb (DEFAULT_TABLE table=test-table [id=59d7a6fe1ea443d08973bb8baf5797e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:11.559409 8267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441575de489a4660bce5a77bd0e73bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:11.576118 8502 tablet_bootstrap.cc:492] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe: Bootstrap starting.
I20260927 06:55:11.577922 8502 tablet_bootstrap.cc:654] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:11.580174 8501 tablet_bootstrap.cc:492] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a: Bootstrap starting.
I20260927 06:55:11.582358 8501 tablet_bootstrap.cc:654] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:11.587203 8504 tablet_bootstrap.cc:492] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f: Bootstrap starting.
I20260927 06:55:11.589246 8504 tablet_bootstrap.cc:654] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:11.591727 8502 tablet_bootstrap.cc:492] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe: No bootstrap required, opened a new log
I20260927 06:55:11.591914 8502 ts_tablet_manager.cc:1397] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.007s
I20260927 06:55:11.592890 8502 raft_consensus.cc:348] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.594326 8502 raft_consensus.cc:374] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:11.594455 8502 raft_consensus.cc:729] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 366565fc5a9a4d3d806537f3bd3d9afe, State: Initialized, Role: FOLLOWER
I20260927 06:55:11.594784 8502 consensus_queue.cc:260] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.595435 8502 ts_tablet_manager.cc:1428] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 06:55:11.598543 8560 ts_tablet_manager.cc:1918] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:55:11.599826 8560 log.cc:1196] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/13ad6c06e33b4ac88a96f40c7166a076
I20260927 06:55:11.600333 8560 ts_tablet_manager.cc:1939] T 13ad6c06e33b4ac88a96f40c7166a076 P 366565fc5a9a4d3d806537f3bd3d9afe: Deleting consensus metadata
I20260927 06:55:11.601404 8562 ts_tablet_manager.cc:1905] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:11.606503 8562 ts_tablet_manager.cc:1918] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:55:11.606657 8562 log.cc:1196] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/13ad6c06e33b4ac88a96f40c7166a076
I20260927 06:55:11.607153 8562 ts_tablet_manager.cc:1939] T 13ad6c06e33b4ac88a96f40c7166a076 P df994aaed44b4eda91d639dde8b5528f: Deleting consensus metadata
I20260927 06:55:11.608347 7979 catalog_manager.cc:4841] TS 366565fc5a9a4d3d806537f3bd3d9afe (127.0.97.1:34427): tablet 13ad6c06e33b4ac88a96f40c7166a076 (table test-table [id=9d9f312a5001456dbefd2c6a0bf8f9ff]) successfully deleted
I20260927 06:55:11.608628 7976 catalog_manager.cc:4841] TS df994aaed44b4eda91d639dde8b5528f (127.0.97.2:35641): tablet 13ad6c06e33b4ac88a96f40c7166a076 (table test-table [id=9d9f312a5001456dbefd2c6a0bf8f9ff]) successfully deleted
I20260927 06:55:11.611964 8504 tablet_bootstrap.cc:492] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f: No bootstrap required, opened a new log
I20260927 06:55:11.612056 8501 tablet_bootstrap.cc:492] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a: No bootstrap required, opened a new log
I20260927 06:55:11.612232 8501 ts_tablet_manager.cc:1397] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.008s
I20260927 06:55:11.612239 8504 ts_tablet_manager.cc:1397] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.007s
I20260927 06:55:11.613086 8501 raft_consensus.cc:348] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.613060 8504 raft_consensus.cc:348] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.613327 8501 raft_consensus.cc:374] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:11.613405 8501 raft_consensus.cc:729] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deb2661436744881b57bddbd30a1950a, State: Initialized, Role: FOLLOWER
I20260927 06:55:11.613523 8504 raft_consensus.cc:374] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:11.613656 8504 raft_consensus.cc:729] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df994aaed44b4eda91d639dde8b5528f, State: Initialized, Role: FOLLOWER
I20260927 06:55:11.613691 8501 consensus_queue.cc:260] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.614102 8504 consensus_queue.cc:260] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.616200 8501 ts_tablet_manager.cc:1428] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260927 06:55:11.616889 8504 ts_tablet_manager.cc:1428] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 06:55:11.726292 8507 raft_consensus.cc:482] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:55:11.726517 8507 raft_consensus.cc:504] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.727895 8507 leader_election.cc:288] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 366565fc5a9a4d3d806537f3bd3d9afe (127.0.97.1:34427), df994aaed44b4eda91d639dde8b5528f (127.0.97.2:35641)
I20260927 06:55:11.732337 8556 raft_consensus.cc:482] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:55:11.732548 8556 raft_consensus.cc:504] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.733180 8556 leader_election.cc:288] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 366565fc5a9a4d3d806537f3bd3d9afe (127.0.97.1:34427), deb2661436744881b57bddbd30a1950a (127.0.97.3:39213)
I20260927 06:55:11.734526 8426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441575de489a4660bce5a77bd0e73bbb" candidate_uuid: "df994aaed44b4eda91d639dde8b5528f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb2661436744881b57bddbd30a1950a" is_pre_election: true
I20260927 06:55:11.734766 8426 raft_consensus.cc:2454] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df994aaed44b4eda91d639dde8b5528f in term 0.
I20260927 06:55:11.735292 8221 leader_election.cc:302] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: deb2661436744881b57bddbd30a1950a, df994aaed44b4eda91d639dde8b5528f; no voters:
I20260927 06:55:11.733946 8153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441575de489a4660bce5a77bd0e73bbb" candidate_uuid: "df994aaed44b4eda91d639dde8b5528f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" is_pre_election: true
I20260927 06:55:11.735860 8153 raft_consensus.cc:2454] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df994aaed44b4eda91d639dde8b5528f in term 0.
I20260927 06:55:11.736558 8556 raft_consensus.cc:2781] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:55:11.736670 8556 raft_consensus.cc:482] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:55:11.736795 8556 raft_consensus.cc:3037] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:11.737142 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:11.728267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60537 (local address 127.0.97.2:35641)
0927 06:55:11.728435 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:11.728439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:11.730236 (+ 1797us) server_negotiation.cc:407] Connection header received
0927 06:55:11.730393 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:11.730399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:11.730476 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:11.730572 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:11.731172 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.732138 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.734681 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.735939 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.736034 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.736127 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:11.736677 (+ 550us) server_negotiation.cc:299] Negotiation successful
0927 06:55:11.736860 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 06:55:11.737520 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:11.728305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54775 (local address 127.0.97.1:34427)
0927 06:55:11.728474 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:11.728480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:11.729629 (+ 1149us) server_negotiation.cc:407] Connection header received
0927 06:55:11.729906 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:11.729912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:11.729994 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:11.730086 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:11.730890 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.732002 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.735073 (+ 3071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.736393 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.736484 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.736574 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:11.737133 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 06:55:11.737271 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 06:55:11.737768 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:11.728716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35641 (local address 127.0.97.3:60537)
0927 06:55:11.730123 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:11.730144 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:11.730266 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:11.730593 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:11.730601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:11.730626 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:11.731024 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:11.731035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.732268 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.732274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:11.734481 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:11.734495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.736306 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.736312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:11.736473 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.736506 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 06:55:11.737274 (+ 768us) client_negotiation.cc:240] Negotiation successful
0927 06:55:11.737403 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1264,"mutex_wait_us":49,"thread_start_us":1045,"threads_started":1}
I20260927 06:55:11.738258 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:11.728714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34427 (local address 127.0.97.3:54775)
0927 06:55:11.729493 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:11.729518 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:11.729697 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:11.730292 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:11.730296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:11.730321 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:11.730706 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:11.730718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.732150 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.732156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:11.734894 (+ 2738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:11.734910 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.736772 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.736779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:11.736952 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.736985 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 06:55:11.737714 (+ 729us) client_negotiation.cc:240] Negotiation successful
0927 06:55:11.737873 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":4,"thread_start_us":444,"threads_started":1}
I20260927 06:55:11.738878 8153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441575de489a4660bce5a77bd0e73bbb" candidate_uuid: "deb2661436744881b57bddbd30a1950a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" is_pre_election: true
I20260927 06:55:11.739138 8153 raft_consensus.cc:2454] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate deb2661436744881b57bddbd30a1950a in term 0.
I20260927 06:55:11.739574 8361 leader_election.cc:302] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366565fc5a9a4d3d806537f3bd3d9afe, deb2661436744881b57bddbd30a1950a; no voters:
I20260927 06:55:11.739595 8556 raft_consensus.cc:504] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.739943 8507 raft_consensus.cc:2781] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:55:11.740051 8507 raft_consensus.cc:482] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:55:11.740126 8507 raft_consensus.cc:3037] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:11.740178 8556 leader_election.cc:288] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [CANDIDATE]: Term 1 election: Requested vote from peers 366565fc5a9a4d3d806537f3bd3d9afe (127.0.97.1:34427), deb2661436744881b57bddbd30a1950a (127.0.97.3:39213)
I20260927 06:55:11.740509 8153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441575de489a4660bce5a77bd0e73bbb" candidate_uuid: "df994aaed44b4eda91d639dde8b5528f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "366565fc5a9a4d3d806537f3bd3d9afe"
I20260927 06:55:11.740703 8153 raft_consensus.cc:3037] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:11.741341 8293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441575de489a4660bce5a77bd0e73bbb" candidate_uuid: "deb2661436744881b57bddbd30a1950a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df994aaed44b4eda91d639dde8b5528f" is_pre_election: true
I20260927 06:55:11.741636 8293 raft_consensus.cc:2379] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate deb2661436744881b57bddbd30a1950a in current term 1: Already voted for candidate df994aaed44b4eda91d639dde8b5528f in this term.
I20260927 06:55:11.742221 8426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441575de489a4660bce5a77bd0e73bbb" candidate_uuid: "df994aaed44b4eda91d639dde8b5528f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb2661436744881b57bddbd30a1950a"
I20260927 06:55:11.743090 8507 raft_consensus.cc:504] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.743530 8426 raft_consensus.cc:2379] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df994aaed44b4eda91d639dde8b5528f in current term 1: Already voted for candidate deb2661436744881b57bddbd30a1950a in this term.
I20260927 06:55:11.743664 8153 raft_consensus.cc:2454] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df994aaed44b4eda91d639dde8b5528f in term 1.
I20260927 06:55:11.744058 8222 leader_election.cc:302] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366565fc5a9a4d3d806537f3bd3d9afe, df994aaed44b4eda91d639dde8b5528f; no voters:
I20260927 06:55:11.744030 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441575de489a4660bce5a77bd0e73bbb" candidate_uuid: "deb2661436744881b57bddbd30a1950a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "366565fc5a9a4d3d806537f3bd3d9afe"
I20260927 06:55:11.744297 8154 raft_consensus.cc:2379] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate deb2661436744881b57bddbd30a1950a in current term 1: Already voted for candidate df994aaed44b4eda91d639dde8b5528f in this term.
I20260927 06:55:11.744545 8556 raft_consensus.cc:2781] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:55:11.744969 8507 leader_election.cc:288] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [CANDIDATE]: Term 1 election: Requested vote from peers 366565fc5a9a4d3d806537f3bd3d9afe (127.0.97.1:34427), df994aaed44b4eda91d639dde8b5528f (127.0.97.2:35641)
I20260927 06:55:11.745074 8556 raft_consensus.cc:686] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [term 1 LEADER]: Becoming Leader. State: Replica: df994aaed44b4eda91d639dde8b5528f, State: Running, Role: LEADER
I20260927 06:55:11.745139 8293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441575de489a4660bce5a77bd0e73bbb" candidate_uuid: "deb2661436744881b57bddbd30a1950a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df994aaed44b4eda91d639dde8b5528f"
I20260927 06:55:11.745347 8556 consensus_queue.cc:237] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } }
I20260927 06:55:11.745729 8358 leader_election.cc:302] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: deb2661436744881b57bddbd30a1950a; no voters: 366565fc5a9a4d3d806537f3bd3d9afe, df994aaed44b4eda91d639dde8b5528f
I20260927 06:55:11.745999 8507 raft_consensus.cc:2726] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 06:55:11.748090 7992 catalog_manager.cc:5495] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f reported cstate change: term changed from 0 to 1, leader changed from <none> to df994aaed44b4eda91d639dde8b5528f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "df994aaed44b4eda91d639dde8b5528f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df994aaed44b4eda91d639dde8b5528f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 } health_report { overall_health: UNKNOWN } } }
I20260927 06:55:11.774758 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:11.770053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.0.97.2:35641)
0927 06:55:11.770189 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:11.770195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:11.770349 (+ 154us) server_negotiation.cc:407] Connection header received
0927 06:55:11.770562 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:11.770568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:11.770637 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:11.770727 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:55:11.771101 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.771792 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.773040 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.773514 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.773643 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.774091 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:55:11.774185 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:55:11.774279 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:11.774434 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 06:55:11.774505 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 06:55:11.782866 8426 raft_consensus.cc:1261] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Refusing update from remote peer df994aaed44b4eda91d639dde8b5528f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:55:11.782883 8154 raft_consensus.cc:1261] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 FOLLOWER]: Refusing update from remote peer df994aaed44b4eda91d639dde8b5528f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:55:11.783439 8556 consensus_queue.cc:1035] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [LEADER]: Connected to new peer: Peer: permanent_uuid: "366565fc5a9a4d3d806537f3bd3d9afe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:55:11.783661 8512 consensus_queue.cc:1035] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [LEADER]: Connected to new peer: Peer: permanent_uuid: "deb2661436744881b57bddbd30a1950a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:55:11.825423 7992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:55:11.825644 7992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:55:11.828414 7992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3dc9badb81654bed99e411125dbe582a: Sending DeleteTablet for 3 replicas of tablet 441575de489a4660bce5a77bd0e73bbb
I20260927 06:55:11.829290 8268 tablet_service.cc:1514] Processing DeleteTablet for tablet 441575de489a4660bce5a77bd0e73bbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49202
I20260927 06:55:11.829445 8407 tablet_service.cc:1514] Processing DeleteTablet for tablet 441575de489a4660bce5a77bd0e73bbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50386
I20260927 06:55:11.829424 8129 tablet_service.cc:1514] Processing DeleteTablet for tablet 441575de489a4660bce5a77bd0e73bbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44110
I20260927 06:55:11.829557 8562 tablet_replica.cc:330] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f: stopping tablet replica
I20260927 06:55:11.829684 8561 tablet_replica.cc:330] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a: stopping tablet replica
I20260927 06:55:11.829694 8560 tablet_replica.cc:330] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe: stopping tablet replica
I20260927 06:55:11.829730 8562 raft_consensus.cc:2229] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:55:11.829847 8560 raft_consensus.cc:2229] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:55:11.829841 8561 raft_consensus.cc:2229] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:55:11.830053 8560 raft_consensus.cc:2258] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:11.830053 8561 raft_consensus.cc:2258] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:11.830052 8562 raft_consensus.cc:2258] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:11.831187 8560 ts_tablet_manager.cc:1905] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:11.831172 8562 ts_tablet_manager.cc:1905] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:11.831224 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 8063
I20260927 06:55:11.834163 8561 ts_tablet_manager.cc:1905] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:11.837666 8562 ts_tablet_manager.cc:1918] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:55:11.837790 8562 log.cc:1196] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/441575de489a4660bce5a77bd0e73bbb
I20260927 06:55:11.837839 8560 ts_tablet_manager.cc:1918] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:55:11.838267 8562 ts_tablet_manager.cc:1939] T 441575de489a4660bce5a77bd0e73bbb P df994aaed44b4eda91d639dde8b5528f: Deleting consensus metadata
I20260927 06:55:11.839325 7976 catalog_manager.cc:4841] TS df994aaed44b4eda91d639dde8b5528f (127.0.97.2:35641): tablet 441575de489a4660bce5a77bd0e73bbb (table test-table [id=59d7a6fe1ea443d08973bb8baf5797e4]) successfully deleted
I20260927 06:55:11.840854 8561 ts_tablet_manager.cc:1918] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:55:11.840984 8561 log.cc:1196] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/441575de489a4660bce5a77bd0e73bbb
I20260927 06:55:11.841400 8561 ts_tablet_manager.cc:1939] T 441575de489a4660bce5a77bd0e73bbb P deb2661436744881b57bddbd30a1950a: Deleting consensus metadata
I20260927 06:55:11.842525 7978 catalog_manager.cc:4841] TS deb2661436744881b57bddbd30a1950a (127.0.97.3:39213): tablet 441575de489a4660bce5a77bd0e73bbb (table test-table [id=59d7a6fe1ea443d08973bb8baf5797e4]) successfully deleted
I20260927 06:55:11.837984 8560 log.cc:1196] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/441575de489a4660bce5a77bd0e73bbb
I20260927 06:55:11.843713 8560 ts_tablet_manager.cc:1939] T 441575de489a4660bce5a77bd0e73bbb P 366565fc5a9a4d3d806537f3bd3d9afe: Deleting consensus metadata
I20260927 06:55:11.844594 7979 catalog_manager.cc:4841] TS 366565fc5a9a4d3d806537f3bd3d9afe (127.0.97.1:34427): tablet 441575de489a4660bce5a77bd0e73bbb (table test-table [id=59d7a6fe1ea443d08973bb8baf5797e4]) successfully deleted
Sep 27 06:55:11 dist-test-slave-f892 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260927 06:55:11.861861 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:11.838715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44134 (local address 127.0.97.1:34427)
0927 06:55:11.838884 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:11.838891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:11.846512 (+ 7621us) server_negotiation.cc:407] Connection header received
0927 06:55:11.846689 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:11.846692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:11.846759 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:11.846843 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:11.848450 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.849190 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.850966 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:11.851358 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:11.851451 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:11.855948 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:11.855960 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:11.855981 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:11.856019 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:11.859224 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:11.859862 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:11.859870 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:11.859874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:11.860013 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:11.860360 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:11.860365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:11.860369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:11.860716 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:11.861020 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:11.861277 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 06:55:11.861497 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 06:55:11.862536 8194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:11.986706 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 8063
I20260927 06:55:12.007965 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 8203
Sep 27 06:55:12 dist-test-slave-f892 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260927 06:55:12.021891 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:12.008820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49232 (local address 127.0.97.2:35641)
0927 06:55:12.008976 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:12.008981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:12.008998 (+ 17us) server_negotiation.cc:407] Connection header received
0927 06:55:12.009050 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:12.009054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:12.009116 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:12.009197 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:12.010341 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:12.011032 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:12.012227 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:12.012617 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:12.012702 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:12.016656 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:12.016668 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:12.016683 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:12.016728 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:12.019588 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:12.020227 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:12.020234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:12.020236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:12.020352 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:12.020628 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:12.020632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:12.020634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:12.020980 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:12.021187 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:12.021394 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 06:55:12.021543 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 06:55:12.022594 8333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:12.165325 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 8203
I20260927 06:55:12.190287 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 8342
Sep 27 06:55:12 dist-test-slave-f892 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260927 06:55:12.205650 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:12.190997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50388 (local address 127.0.97.3:39213)
0927 06:55:12.191481 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:12.191486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:12.191501 (+ 15us) server_negotiation.cc:407] Connection header received
0927 06:55:12.191549 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:12.191552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:12.191605 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:12.191694 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:12.192906 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:12.193584 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:12.194797 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:12.195182 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:12.195269 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:12.199365 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:12.199381 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:12.199402 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:12.199437 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:12.203070 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:12.203659 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:12.203669 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:12.203673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:12.203840 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:12.204148 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:12.204153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:12.204157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:12.204582 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:12.204866 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:12.205053 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 06:55:12.205290 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":280,"threads_started":1}
I20260927 06:55:12.206398 8472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:12.301767 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 8342
I20260927 06:55:12.319064 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 7960
I20260927 06:55:12.320225 8032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:12.432967 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 7960
2026-09-27T06:55:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5574 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:55:12 dist-test-slave-f892 krb5kdc[8605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:55:12 dist-test-slave-f892 krb5kdc[8605](info): set up 2 sockets
Sep 27 06:55:12 dist-test-slave-f892 krb5kdc[8605](info): commencing operation
krb5kdc: starting...
W20260927 06:55:13.550877 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.005s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:55:13 dist-test-slave-f892 krb5kdc[8605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:55:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:55:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:55:13.761930 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33061
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:55:14.075150 8618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:14.075395 8618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:14.075472 8618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:14.082108 8618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:55:14.082214 8618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:14.082255 8618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:14.082309 8618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:55:14.082356 8618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:55:14.092377 8618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:14.093583 8618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:14.094902 8618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:14.106307 8623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:14.106287 8626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:14.106812 8618 server_base.cc:1034] running on GCE node
W20260927 06:55:14.106312 8624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:14.107681 8618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:14.108886 8618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:14.110121 8618 hybrid_clock.cc:648] HybridClock initialized: now 1790492114110086 us; error 28 us; skew 500 ppm
I20260927 06:55:14.110466 8618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T06:55:14Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1745 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:55:14 dist-test-slave-f892 krb5kdc[8632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:55:14 dist-test-slave-f892 krb5kdc[8632](info): set up 2 sockets
Sep 27 06:55:14 dist-test-slave-f892 krb5kdc[8632](info): commencing operation
krb5kdc: starting...
W20260927 06:55:15.274912 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:55:15 dist-test-slave-f892 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:55:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:55:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:55:15.491550 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42163
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:55:15.806996 8645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:15.807308 8645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:15.807421 8645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:15.814625 8645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:55:15.814738 8645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:15.814785 8645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:15.814841 8645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:55:15.814889 8645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:55:15.824719 8645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:15.825965 8645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:15.827263 8645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:15.838485 8650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:15.838519 8651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:15.838603 8653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:15.839470 8645 server_base.cc:1034] running on GCE node
I20260927 06:55:15.840253 8645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:15.841573 8645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:15.842795 8645 hybrid_clock.cc:648] HybridClock initialized: now 1790492115842756 us; error 45 us; skew 500 ppm
I20260927 06:55:15.843143 8645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:15 dist-test-slave-f892 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:15.846292 8645 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:55:15.847769 8645 webserver.cc:460] Webserver started at http://127.0.97.62:41159/ using document root <none> and password file <none>
I20260927 06:55:15.848404 8645 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:15.848505 8645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:15.848753 8645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:15.851697 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "e8437ecdde984e5b855d441df6a8e920"
format_stamp: "Formatted at 2026-09-27 06:55:15 on dist-test-slave-f892"
server_key: "3a76a5b111769d0109182bc1676a2579"
server_key_iv: "c4ef201a7a16bd1460f772a61946bdb4"
server_key_version: "encryptionkey@0"
I20260927 06:55:15.852475 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "e8437ecdde984e5b855d441df6a8e920"
format_stamp: "Formatted at 2026-09-27 06:55:15 on dist-test-slave-f892"
server_key: "3a76a5b111769d0109182bc1676a2579"
server_key_iv: "c4ef201a7a16bd1460f772a61946bdb4"
server_key_version: "encryptionkey@0"
I20260927 06:55:15.857342 8645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 06:55:15.860577 8660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:15.861822 8645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 06:55:15.861996 8645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "e8437ecdde984e5b855d441df6a8e920"
format_stamp: "Formatted at 2026-09-27 06:55:15 on dist-test-slave-f892"
server_key: "3a76a5b111769d0109182bc1676a2579"
server_key_iv: "c4ef201a7a16bd1460f772a61946bdb4"
server_key_version: "encryptionkey@0"
I20260927 06:55:15.862253 8645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:15.889261 8645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:15.893651 8645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:15.893954 8645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:15.911659 8645 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42163
I20260927 06:55:15.911690 8711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42163 every 8 connection(s)
I20260927 06:55:15.913411 8645 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:55:15.916936 8712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:15.921620 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 8645
I20260927 06:55:15.921828 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:55:15.922676 388 external_mini_cluster.cc:1428] Setting key 105c8f9b3b5cb72b233201eb4d400f53
I20260927 06:55:15.931449 8712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920: Bootstrap starting.
I20260927 06:55:15.935134 8712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:15.936442 8712 log.cc:824] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920: Log is configured to *not* fsync() on all Append() calls
Sep 27 06:55:15 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:55:15.939754 8712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920: No bootstrap required, opened a new log
I20260927 06:55:15.946416 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:15.924826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38318 (local address 127.0.97.62:42163)
0927 06:55:15.925666 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:15.925677 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:15.925719 (+ 42us) server_negotiation.cc:407] Connection header received
0927 06:55:15.927366 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:15.927389 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:15.927803 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:15.928134 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:15.929791 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:15.931101 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:15.932760 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:15.933418 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:15.933542 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:15.938380 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:15.938389 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:15.938408 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:15.938453 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:15.942221 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:15.942911 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:15.942924 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:15.942932 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:15.943129 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:15.943461 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:15.943466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:15.943470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:15.944145 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:15.944400 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:15.944685 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 06:55:15.945121 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":419,"threads_started":1}
I20260927 06:55:15.949174 8712 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8437ecdde984e5b855d441df6a8e920" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42163 } }
I20260927 06:55:15.949468 8712 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:15.949549 8712 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8437ecdde984e5b855d441df6a8e920, State: Initialized, Role: FOLLOWER
I20260927 06:55:15.949985 8712 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8437ecdde984e5b855d441df6a8e920" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42163 } }
I20260927 06:55:15.950237 8712 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:55:15.950328 8712 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:55:15.950453 8712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:15.952412 8712 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8437ecdde984e5b855d441df6a8e920" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42163 } }
I20260927 06:55:15.952806 8712 leader_election.cc:302] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8437ecdde984e5b855d441df6a8e920; no voters:
I20260927 06:55:15.953295 8712 leader_election.cc:288] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:55:15.953480 8717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:55:15.954564 8717 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [term 1 LEADER]: Becoming Leader. State: Replica: e8437ecdde984e5b855d441df6a8e920, State: Running, Role: LEADER
I20260927 06:55:15.955040 8717 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8437ecdde984e5b855d441df6a8e920" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42163 } }
I20260927 06:55:15.955335 8712 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:55:15.958146 8719 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8437ecdde984e5b855d441df6a8e920. Latest consensus state: current_term: 1 leader_uuid: "e8437ecdde984e5b855d441df6a8e920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8437ecdde984e5b855d441df6a8e920" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42163 } } }
I20260927 06:55:15.958477 8719 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:15.957974 8718 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8437ecdde984e5b855d441df6a8e920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8437ecdde984e5b855d441df6a8e920" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42163 } } }
I20260927 06:55:15.958874 8718 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:15.960140 8725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:55:15.964632 8725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:55:15.971316 8734 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 06:55:15.973125 8725 catalog_manager.cc:1347] Generated new cluster ID: abc70ee5252f4711b6217c839b8c90fe
I20260927 06:55:15.973224 8725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:55:16.010051 8725 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:55:16.011394 8725 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:55:16.018833 8725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920: Generated new TSK 0
I20260927 06:55:16.019932 8725 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 06:55:16.079936 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42163
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:55:16.402596 8738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:16.402915 8738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:16.403083 8738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:16.410813 8738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:16.410964 8738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:16.411131 8738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:55:16.420929 8738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:16.422367 8738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:16.423650 8738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:16.438220 8744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:16.438293 8743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:16.438264 8746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:16.439124 8738 server_base.cc:1034] running on GCE node
I20260927 06:55:16.439757 8738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:16.440768 8738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:16.441957 8738 hybrid_clock.cc:648] HybridClock initialized: now 1790492116441898 us; error 51 us; skew 500 ppm
I20260927 06:55:16.442299 8738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:16 dist-test-slave-f892 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:16.445307 8738 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:55:16.446724 8738 webserver.cc:460] Webserver started at http://127.0.97.1:38845/ using document root <none> and password file <none>
I20260927 06:55:16.447496 8738 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:16.447597 8738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:16.447849 8738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:16.450786 8738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "397fc859055f45fe8da8b93e1ac25ef2"
format_stamp: "Formatted at 2026-09-27 06:55:16 on dist-test-slave-f892"
server_key: "5e3dd6aaced60fbcb48733d8fbd4bcad"
server_key_iv: "5070ccfab6ba6817a03af60e0a21488f"
server_key_version: "encryptionkey@0"
I20260927 06:55:16.451491 8738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "397fc859055f45fe8da8b93e1ac25ef2"
format_stamp: "Formatted at 2026-09-27 06:55:16 on dist-test-slave-f892"
server_key: "5e3dd6aaced60fbcb48733d8fbd4bcad"
server_key_iv: "5070ccfab6ba6817a03af60e0a21488f"
server_key_version: "encryptionkey@0"
I20260927 06:55:16.456298 8738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 06:55:16.459558 8753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:16.461011 8738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 06:55:16.461174 8738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "397fc859055f45fe8da8b93e1ac25ef2"
format_stamp: "Formatted at 2026-09-27 06:55:16 on dist-test-slave-f892"
server_key: "5e3dd6aaced60fbcb48733d8fbd4bcad"
server_key_iv: "5070ccfab6ba6817a03af60e0a21488f"
server_key_version: "encryptionkey@0"
I20260927 06:55:16.461381 8738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:16.500835 8738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:16.504514 8738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:16.504742 8738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:16.505574 8738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:16.506778 8738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:16.506844 8738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:16.506947 8738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:16.506999 8738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:16.538504 8738 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45723
I20260927 06:55:16.538580 8865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45723 every 8 connection(s)
I20260927 06:55:16.539781 8738 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:55:16.542088 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 8738
I20260927 06:55:16.542289 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:55:16.543298 388 external_mini_cluster.cc:1428] Setting key 7417fc80e4fc25969ead19f2d1fe9687
Sep 27 06:55:16 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:55:16.558739 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:16.542266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39867 (local address 127.0.97.62:42163)
0927 06:55:16.544208 (+ 1942us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:16.544216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:16.544237 (+ 21us) server_negotiation.cc:407] Connection header received
0927 06:55:16.545542 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:16.545547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:16.545622 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:16.545742 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:16.547151 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:16.547860 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:16.549343 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:16.549759 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:16.549887 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:16.553314 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:16.553319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:16.553323 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:16.553349 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:16.555869 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:16.556616 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:16.556623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:16.556630 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:16.556728 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:16.557135 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:16.557140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:16.557143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:16.557379 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:16.557549 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:16.558209 (+ 660us) server_negotiation.cc:299] Negotiation successful
0927 06:55:16.558410 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1824,"thread_start_us":1727,"threads_started":1}
I20260927 06:55:16.559609 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:16.542761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:39867)
0927 06:55:16.543535 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:16.543584 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:16.545299 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:16.545952 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:16.545959 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:16.546503 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:16.547001 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:16.547016 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:16.548027 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:16.548032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:16.549203 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:16.549213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:16.549967 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:16.549975 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:16.550222 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:16.550863 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:16.550885 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:16.553111 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:16.556029 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:16.556038 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:16.556054 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:16.556440 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:16.556828 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:16.556834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:16.556837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:16.557024 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:16.557506 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:16.557513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:16.557808 (+ 295us) client_negotiation.cc:769] Sending connection context
0927 06:55:16.558084 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 06:55:16.558557 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":332,"threads_started":1}
I20260927 06:55:16.561092 8866 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42163
I20260927 06:55:16.561422 8866 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:16.562332 8866 heartbeater.cc:511] Master 127.0.97.62:42163 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 06:55:16.564648 8677 ts_manager.cc:194] Registered new tserver with Master: 397fc859055f45fe8da8b93e1ac25ef2 (127.0.97.1:45723)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 06:55:16.566411 8677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 06:55:16.590669 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42163
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:55:16.907088 8872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:16.907358 8872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:16.907533 8872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:16.914583 8872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:16.914677 8872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:16.914839 8872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:55:16.924448 8872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:16.925711 8872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:16.927088 8872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:16.940085 8878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:16.940140 8880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:16.940166 8877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:16.941181 8872 server_base.cc:1034] running on GCE node
I20260927 06:55:16.941776 8872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:16.942688 8872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:16.943907 8872 hybrid_clock.cc:648] HybridClock initialized: now 1790492116943868 us; error 50 us; skew 500 ppm
I20260927 06:55:16.944192 8872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:16 dist-test-slave-f892 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:16.947165 8872 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:55:16.948514 8872 webserver.cc:460] Webserver started at http://127.0.97.2:34551/ using document root <none> and password file <none>
I20260927 06:55:16.949113 8872 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:16.949246 8872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:16.949532 8872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:16.952246 8872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "bb55c5777ef44d2cb51ce25aa1e5b9a2"
format_stamp: "Formatted at 2026-09-27 06:55:16 on dist-test-slave-f892"
server_key: "c565c80ad1159c9cdf8439aa9c96a663"
server_key_iv: "4abe31d7b529ead19ec3072e6f545c9f"
server_key_version: "encryptionkey@0"
I20260927 06:55:16.952973 8872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bb55c5777ef44d2cb51ce25aa1e5b9a2"
format_stamp: "Formatted at 2026-09-27 06:55:16 on dist-test-slave-f892"
server_key: "c565c80ad1159c9cdf8439aa9c96a663"
server_key_iv: "4abe31d7b529ead19ec3072e6f545c9f"
server_key_version: "encryptionkey@0"
I20260927 06:55:16.957605 8872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 06:55:16.960856 8887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:16.962143 8872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 06:55:16.962319 8872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "bb55c5777ef44d2cb51ce25aa1e5b9a2"
format_stamp: "Formatted at 2026-09-27 06:55:16 on dist-test-slave-f892"
server_key: "c565c80ad1159c9cdf8439aa9c96a663"
server_key_iv: "4abe31d7b529ead19ec3072e6f545c9f"
server_key_version: "encryptionkey@0"
I20260927 06:55:16.962499 8872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:16.976069 8872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:16.980183 8872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:16.980470 8872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:16.981343 8872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:16.982643 8872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:16.982712 8872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:16.982854 8872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:16.982915 8872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:17.014030 8872 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35307
I20260927 06:55:17.014137 8999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35307 every 8 connection(s)
I20260927 06:55:17.015232 8872 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:55:17.019542 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 8872
I20260927 06:55:17.019721 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:55:17.020136 388 external_mini_cluster.cc:1428] Setting key ef4fe220fb3fb6b6f5ae1380b6bc8c49
Sep 27 06:55:17 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:55:17.036257 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:17.017843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52961 (local address 127.0.97.62:42163)
0927 06:55:17.018085 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:17.018090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:17.019162 (+ 1072us) server_negotiation.cc:407] Connection header received
0927 06:55:17.021230 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:17.021235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:17.021319 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:17.021425 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:17.023031 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:17.023705 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:17.025200 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:17.025606 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:17.025737 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:17.029359 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:17.029368 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:17.029376 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:17.029419 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:17.033006 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:17.033698 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:17.033707 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:17.033713 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:17.033868 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:17.034306 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:17.034313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:17.034319 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:17.034688 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:17.034867 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:17.035547 (+ 680us) server_negotiation.cc:299] Negotiation successful
0927 06:55:17.035792 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 06:55:17.036805 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:17.018327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:52961)
0927 06:55:17.018997 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:17.019039 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:17.020970 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:17.021670 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:17.021681 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:17.022333 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:17.022871 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:17.022888 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:17.023849 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:17.023855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:17.025057 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:17.025069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:17.025818 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:17.025826 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:17.026119 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:17.026787 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:17.026807 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:17.029121 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:17.033140 (+ 4019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:17.033146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:17.033158 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:17.033553 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:17.033970 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:17.033973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:17.033975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:17.034190 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:17.034821 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:17.034829 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:17.035197 (+ 368us) client_negotiation.cc:769] Sending connection context
0927 06:55:17.035406 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 06:55:17.035755 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":276,"threads_started":1}
I20260927 06:55:17.038214 9000 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42163
I20260927 06:55:17.038568 9000 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:17.039448 9000 heartbeater.cc:511] Master 127.0.97.62:42163 requested a full tablet report, sending...
I20260927 06:55:17.040957 8677 ts_manager.cc:194] Registered new tserver with Master: bb55c5777ef44d2cb51ce25aa1e5b9a2 (127.0.97.2:35307)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 06:55:17.041869 8677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 06:55:17.068929 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42163
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:55:17.379122 9005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:17.379410 9005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:17.379567 9005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:17.386456 9005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:17.386585 9005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:17.386765 9005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:55:17.395964 9005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:17.397161 9005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:17.398521 9005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:17.411329 9011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:17.411369 9013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:17.411293 9010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:17.412249 9005 server_base.cc:1034] running on GCE node
I20260927 06:55:17.412803 9005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:17.413712 9005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:17.414929 9005 hybrid_clock.cc:648] HybridClock initialized: now 1790492117414892 us; error 50 us; skew 500 ppm
I20260927 06:55:17.415251 9005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:17 dist-test-slave-f892 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:17.418327 9005 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:55:17.419742 9005 webserver.cc:460] Webserver started at http://127.0.97.3:33885/ using document root <none> and password file <none>
I20260927 06:55:17.420365 9005 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:17.420461 9005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:17.420708 9005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:17.423555 9005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "1f891814adb14482bedeeb5d6de0f5f4"
format_stamp: "Formatted at 2026-09-27 06:55:17 on dist-test-slave-f892"
server_key: "4aa4b11128a0ab77f0488c33429970e9"
server_key_iv: "95a077b175f296423ac5425e246a4612"
server_key_version: "encryptionkey@0"
I20260927 06:55:17.424297 9005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1f891814adb14482bedeeb5d6de0f5f4"
format_stamp: "Formatted at 2026-09-27 06:55:17 on dist-test-slave-f892"
server_key: "4aa4b11128a0ab77f0488c33429970e9"
server_key_iv: "95a077b175f296423ac5425e246a4612"
server_key_version: "encryptionkey@0"
I20260927 06:55:17.428982 9005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 06:55:17.432273 9020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:17.433533 9005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 06:55:17.433676 9005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "1f891814adb14482bedeeb5d6de0f5f4"
format_stamp: "Formatted at 2026-09-27 06:55:17 on dist-test-slave-f892"
server_key: "4aa4b11128a0ab77f0488c33429970e9"
server_key_iv: "95a077b175f296423ac5425e246a4612"
server_key_version: "encryptionkey@0"
I20260927 06:55:17.433874 9005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:17.452126 9005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:17.456393 9005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:17.456670 9005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:17.457486 9005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:17.458663 9005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:17.458730 9005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:17.458808 9005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:17.458850 9005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:17.490523 9005 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34701
I20260927 06:55:17.490576 9132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34701 every 8 connection(s)
I20260927 06:55:17.491720 9005 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:55:17.496569 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 9005
I20260927 06:55:17.496726 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:55:17.497259 388 external_mini_cluster.cc:1428] Setting key 608e9b3b028a815dda62a61968b35ac3
Sep 27 06:55:17 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:55:17.510358 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:17.494062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46143 (local address 127.0.97.62:42163)
0927 06:55:17.494227 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:17.494232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:17.495374 (+ 1142us) server_negotiation.cc:407] Connection header received
0927 06:55:17.497619 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:17.497624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:17.497700 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:17.497823 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:17.499553 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:17.500443 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:17.501862 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:17.502335 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:17.502429 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:17.505448 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:17.505453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:17.505456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:17.505484 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:17.508026 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:17.508591 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:17.508594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:17.508596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:17.508722 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:17.509059 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:17.509062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:17.509064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:17.509282 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:17.509387 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:17.509927 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 06:55:17.510128 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 06:55:17.511410 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:17.494546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.3:46143)
0927 06:55:17.495224 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:17.495263 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:17.497339 (+ 2076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:17.498057 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:17.498065 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:17.498694 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:17.499385 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:17.499409 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:17.500570 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:17.500574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:17.501741 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:17.501751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:17.502511 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:17.502516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:17.502691 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:17.503301 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:17.503325 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:17.505256 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:17.508131 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:17.508137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:17.508147 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:17.508499 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:17.508809 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:17.508812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:17.508814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:17.508973 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:17.509381 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:17.509386 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:17.509633 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 06:55:17.509854 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 06:55:17.510230 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":292,"threads_started":1}
I20260927 06:55:17.512622 9133 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42163
I20260927 06:55:17.512897 9133 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:17.513525 9133 heartbeater.cc:511] Master 127.0.97.62:42163 requested a full tablet report, sending...
I20260927 06:55:17.514844 8677 ts_manager.cc:194] Registered new tserver with Master: 1f891814adb14482bedeeb5d6de0f5f4 (127.0.97.3:34701)
I20260927 06:55:17.515559 8677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46143
I20260927 06:55:17.524286 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
I20260927 06:55:17.569384 8866 heartbeater.cc:503] Master 127.0.97.62:42163 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 06:55:17.652758 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 8738
Sep 27 06:55:17 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:55:17.669092 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:17.653905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36362 (local address 127.0.97.1:45723)
0927 06:55:17.654548 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:17.654556 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:17.654583 (+ 27us) server_negotiation.cc:407] Connection header received
0927 06:55:17.654653 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:17.654659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:17.654960 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:17.655143 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:17.656810 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:17.657803 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:17.659039 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:17.659607 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:17.659735 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:17.663297 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:17.663307 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:17.663322 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:17.663352 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:17.666526 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:17.667234 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:17.667241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:17.667243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:17.667367 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:17.667757 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:17.667760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:17.667762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:17.668074 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:17.668286 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:17.668512 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 06:55:17.668690 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":353,"threads_started":1}
I20260927 06:55:17.670359 8861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:17.751574 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 8738
I20260927 06:55:17.768225 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 8872
Sep 27 06:55:17 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:55:17.782608 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:17.769090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58890 (local address 127.0.97.2:35307)
0927 06:55:17.769582 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:17.769589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:17.769613 (+ 24us) server_negotiation.cc:407] Connection header received
0927 06:55:17.769669 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:17.769675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:17.769893 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:17.770068 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:17.771139 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:17.772129 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:17.773336 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:17.773835 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:17.773926 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:17.777801 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:17.777811 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:17.777826 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:17.777853 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:17.780398 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:17.780953 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:17.780959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:17.780961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:17.781066 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:17.781330 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:17.781333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:17.781335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:17.781609 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:17.781816 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:17.782072 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 06:55:17.782242 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":271,"threads_started":1}
I20260927 06:55:17.783725 8995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:17.864089 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 8872
I20260927 06:55:17.879549 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 9005
Sep 27 06:55:17 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 06:55:17.894670 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:17.880489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49386 (local address 127.0.97.3:34701)
0927 06:55:17.881037 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:17.881043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:17.881070 (+ 27us) server_negotiation.cc:407] Connection header received
0927 06:55:17.881123 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:17.881129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:17.881322 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:17.881472 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:17.882695 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:17.883724 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:17.884887 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:17.885401 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:17.885493 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:17.889640 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:17.889651 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:17.889665 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:17.889694 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:17.892327 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:17.892885 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:17.892891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:17.892893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:17.892999 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:17.893281 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:17.893284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:17.893286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:17.893592 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:17.893826 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:17.894129 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 06:55:17.894376 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":244,"threads_started":1}
I20260927 06:55:17.895893 9128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:17.982745 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 9005
I20260927 06:55:17.998281 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 8645
I20260927 06:55:17.999307 8707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:18.098171 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 8645
I20260927 06:55:18.117890 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42163
--webserver_interface=127.0.97.62
--webserver_port=41159
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:55:18.425319 9150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:18.425606 9150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:18.425832 9150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:18.433795 9150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:55:18.433887 9150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:18.433981 9150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:18.434108 9150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:55:18.434217 9150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:55:18.444574 9150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=41159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:18.445878 9150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:18.447245 9150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:18.458478 9155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:18.458407 9156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:18.458482 9158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:18.460054 9150 server_base.cc:1034] running on GCE node
I20260927 06:55:18.460857 9150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:18.462198 9150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:18.463444 9150 hybrid_clock.cc:648] HybridClock initialized: now 1790492118463400 us; error 48 us; skew 500 ppm
I20260927 06:55:18.463790 9150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:18 dist-test-slave-f892 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:18.466698 9150 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:55:18.468108 9150 webserver.cc:460] Webserver started at http://127.0.97.62:41159/ using document root <none> and password file <none>
I20260927 06:55:18.468696 9150 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:18.468796 9150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:18.473754 9150 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 06:55:18.476279 9165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:18.477547 9150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 06:55:18.477672 9150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "e8437ecdde984e5b855d441df6a8e920"
format_stamp: "Formatted at 2026-09-27 06:55:15 on dist-test-slave-f892"
server_key: "3a76a5b111769d0109182bc1676a2579"
server_key_iv: "c4ef201a7a16bd1460f772a61946bdb4"
server_key_version: "encryptionkey@0"
I20260927 06:55:18.478575 9150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:18.490270 9150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:18.493736 9150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:18.493935 9150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:18.511642 9150 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42163
I20260927 06:55:18.511677 9216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42163 every 8 connection(s)
I20260927 06:55:18.513010 9150 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:55:18.517987 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 9150
I20260927 06:55:18.518800 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45723
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42163
--webserver_port=38845
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:55:18.519598 9217 sys_catalog.cc:263] Verifying existing consensus state
I20260927 06:55:18.523422 9217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920: Bootstrap starting.
I20260927 06:55:18.541764 9217 log.cc:824] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920: Log is configured to *not* fsync() on all Append() calls
I20260927 06:55:18.550384 9217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 06:55:18.550886 9217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920: Bootstrap complete.
I20260927 06:55:18.560380 9217 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8437ecdde984e5b855d441df6a8e920" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42163 } }
I20260927 06:55:18.561159 9217 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8437ecdde984e5b855d441df6a8e920, State: Initialized, Role: FOLLOWER
I20260927 06:55:18.561748 9217 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8437ecdde984e5b855d441df6a8e920" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42163 } }
I20260927 06:55:18.561971 9217 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:55:18.562088 9217 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:55:18.562274 9217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [term 1 FOLLOWER]: Advancing to term 2
I20260927 06:55:18.565794 9217 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8437ecdde984e5b855d441df6a8e920" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42163 } }
I20260927 06:55:18.566332 9217 leader_election.cc:302] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8437ecdde984e5b855d441df6a8e920; no voters:
I20260927 06:55:18.566984 9217 leader_election.cc:288] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 06:55:18.567116 9221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 06:55:18.568343 9221 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [term 2 LEADER]: Becoming Leader. State: Replica: e8437ecdde984e5b855d441df6a8e920, State: Running, Role: LEADER
I20260927 06:55:18.568910 9221 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8437ecdde984e5b855d441df6a8e920" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42163 } }
I20260927 06:55:18.569196 9217 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:55:18.571497 9222 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e8437ecdde984e5b855d441df6a8e920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8437ecdde984e5b855d441df6a8e920" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42163 } } }
I20260927 06:55:18.571931 9222 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:18.574254 9223 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8437ecdde984e5b855d441df6a8e920. Latest consensus state: current_term: 2 leader_uuid: "e8437ecdde984e5b855d441df6a8e920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8437ecdde984e5b855d441df6a8e920" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42163 } } }
I20260927 06:55:18.574585 9223 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:18.575953 9227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:55:18.581440 9227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:55:18.582903 9227 catalog_manager.cc:1259] Loaded cluster ID: abc70ee5252f4711b6217c839b8c90fe
I20260927 06:55:18.583042 9227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:55:18.586884 9227 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:55:18.588268 9227 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e8437ecdde984e5b855d441df6a8e920: Loaded TSK: 0
I20260927 06:55:18.590101 9227 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 06:55:18.846235 9219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:18.846580 9219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:18.846742 9219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:18.854321 9219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:18.854476 9219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:18.854645 9219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:55:18.864425 9219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38845
--tserver_master_addrs=127.0.97.62:42163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:18.865721 9219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:18.867070 9219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:18.881345 9244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:18.883531 9243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:18.883935 9246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:18.884690 9219 server_base.cc:1034] running on GCE node
I20260927 06:55:18.885528 9219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:18.886675 9219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:18.887991 9219 hybrid_clock.cc:648] HybridClock initialized: now 1790492118887919 us; error 58 us; skew 500 ppm
I20260927 06:55:18.888324 9219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:18 dist-test-slave-f892 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492118, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:18.891482 9219 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 06:55:18.893046 9219 webserver.cc:460] Webserver started at http://127.0.97.1:38845/ using document root <none> and password file <none>
I20260927 06:55:18.893719 9219 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:18.893853 9219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:18.899118 9219 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 06:55:18.901736 9253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:18.903138 9219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 06:55:18.903294 9219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "397fc859055f45fe8da8b93e1ac25ef2"
format_stamp: "Formatted at 2026-09-27 06:55:16 on dist-test-slave-f892"
server_key: "5e3dd6aaced60fbcb48733d8fbd4bcad"
server_key_iv: "5070ccfab6ba6817a03af60e0a21488f"
server_key_version: "encryptionkey@0"
I20260927 06:55:18.904192 9219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:18.929028 9219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:18.933266 9219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:18.933565 9219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:18.934625 9219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:18.935895 9219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:18.936007 9219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:18.936126 9219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:18.936224 9219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:18.968140 9219 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45723
I20260927 06:55:18.968238 9365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45723 every 8 connection(s)
I20260927 06:55:18.969601 9219 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:55:18.979673 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 9219
I20260927 06:55:18.980235 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35307
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42163
--webserver_port=34551
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 06:55:18 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:18 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:18.986994 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:18.972528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:36095)
0927 06:55:18.973381 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:18.973434 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:18.975170 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:18.976496 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:18.976506 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:18.977115 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:18.977685 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:18.977701 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:18.979269 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:18.979277 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:18.980586 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:18.980598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:18.981682 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:18.981708 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:18.981956 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:18.982711 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:18.982745 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:18.985996 (+ 3251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":369,"threads_started":1}
W20260927 06:55:18.987403 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:18.972501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36095 (local address 127.0.97.62:42163)
0927 06:55:18.973336 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:18.973352 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:18.973554 (+ 202us) server_negotiation.cc:407] Connection header received
0927 06:55:18.975718 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:18.975731 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:18.976034 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:18.976366 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:18.977863 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:18.979125 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:18.980720 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:18.981418 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:18.981613 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:18.986427 (+ 4814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36095: BlockingRecv error: recv got EOF from 127.0.97.1:36095 (error 108)
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":344,"threads_started":1}
W20260927 06:55:18.987635 9366 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42163 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:42163: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 27 06:55:18 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:18 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 06:55:18.997282 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:18.988411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35819 (local address 127.0.97.62:42163)
0927 06:55:18.988543 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:18.988547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:18.988560 (+ 13us) server_negotiation.cc:407] Connection header received
0927 06:55:18.988598 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:18.988601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:18.988647 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:18.988718 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:18.989518 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:18.990273 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:18.992228 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:18.992583 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:18.992668 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:18.997124 (+ 4456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35819: BlockingRecv error: recv got EOF from 127.0.97.1:35819 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 06:55:18.997313 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:18.988207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:35819)
0927 06:55:18.988393 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:18.988412 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:18.988501 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:18.988769 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:18.988776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:18.989186 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:18.989387 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:18.989400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:18.990414 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:18.990422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:18.992076 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:18.992092 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:18.992724 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:18.992733 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:18.992949 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:18.993524 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:18.993550 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:18.996966 (+ 3416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
Sep 27 06:55:19 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:19 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:19.005940 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:18.997992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:46291)
0927 06:55:18.998255 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:18.998274 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:18.998359 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:18.998752 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:18.998755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:18.998993 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:18.999112 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:18.999119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.000592 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.000595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:19.001455 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:19.001464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.002480 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.002483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:19.002590 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:19.003115 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:19.003130 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:19.005724 (+ 2594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125}
W20260927 06:55:19.006239 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:18.998219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46291 (local address 127.0.97.62:42163)
0927 06:55:18.998426 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:18.998434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:18.998454 (+ 20us) server_negotiation.cc:407] Connection header received
0927 06:55:18.998509 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:18.998517 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:18.998610 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:18.998736 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:18.999238 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.000462 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.001586 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.002346 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.002497 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:19.005964 (+ 3467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46291: BlockingRecv error: recv got EOF from 127.0.97.1:46291 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 06:55:19.006364 9366 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 06:55:19.297291 9370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:19.297544 9370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:19.297628 9370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:19.304349 9370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:19.304435 9370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:19.304558 9370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:55:19.313983 9370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34551
--tserver_master_addrs=127.0.97.62:42163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:19.315304 9370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:19.316623 9370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:19.329682 9375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:19.329768 9378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:19.329733 9376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:19.330492 9370 server_base.cc:1034] running on GCE node
I20260927 06:55:19.331094 9370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:19.331933 9370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:19.333122 9370 hybrid_clock.cc:648] HybridClock initialized: now 1790492119333052 us; error 67 us; skew 500 ppm
I20260927 06:55:19.333419 9370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:19 dist-test-slave-f892 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492119, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:19.341089 9370 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 06:55:19.342440 9370 webserver.cc:460] Webserver started at http://127.0.97.2:34551/ using document root <none> and password file <none>
I20260927 06:55:19.343088 9370 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:19.343230 9370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:19.348153 9370 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 06:55:19.350576 9385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:19.351856 9370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 06:55:19.352016 9370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "bb55c5777ef44d2cb51ce25aa1e5b9a2"
format_stamp: "Formatted at 2026-09-27 06:55:16 on dist-test-slave-f892"
server_key: "c565c80ad1159c9cdf8439aa9c96a663"
server_key_iv: "4abe31d7b529ead19ec3072e6f545c9f"
server_key_version: "encryptionkey@0"
I20260927 06:55:19.353339 9370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:19.379415 9370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:19.383401 9370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:19.383662 9370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:19.384572 9370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:19.385785 9370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:19.385913 9370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:19.386067 9370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:19.386178 9370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 06:55:19.419554 9370 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35307
I20260927 06:55:19.419641 9497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35307 every 8 connection(s)
I20260927 06:55:19.421020 9370 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:55:19.422758 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 9370
I20260927 06:55:19.423367 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34701
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42163
--webserver_port=33885
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 06:55:19 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:19 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 06:55:19.444106 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:19.423812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45597 (local address 127.0.97.62:42163)
0927 06:55:19.424018 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:19.424025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:19.425095 (+ 1070us) server_negotiation.cc:407] Connection header received
0927 06:55:19.427816 (+ 2721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:19.427820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:19.427881 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:19.427962 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:19.429898 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.430877 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.432328 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.433109 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.433239 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:19.443790 (+ 10551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45597: BlockingRecv error: recv got EOF from 127.0.97.2:45597 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 06:55:19.444487 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:19.424129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:45597)
0927 06:55:19.424913 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:19.424967 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:19.427560 (+ 2593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:19.428219 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:19.428231 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:19.428970 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:19.429718 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:19.429742 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.431030 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.431034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:19.432194 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:19.432204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.433308 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.433312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:19.433496 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:19.440036 (+ 6540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:19.440064 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:19.443466 (+ 3402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":302,"threads_started":1}
W20260927 06:55:19.445207 9498 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42163 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:42163: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 27 06:55:19 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:19 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:19.453554 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:19.445788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:59503)
0927 06:55:19.445927 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:19.445948 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:19.446074 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:19.446514 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:19.446517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:19.446800 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:19.446942 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:19.446950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.447905 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.447910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:19.448910 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:19.448921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.449588 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.449594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:19.449762 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:19.450380 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:19.450396 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:19.453279 (+ 2883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 06:55:19.453637 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:19.445891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59503 (local address 127.0.97.62:42163)
0927 06:55:19.446087 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:19.446092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:19.446113 (+ 21us) server_negotiation.cc:407] Connection header received
0927 06:55:19.446175 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:19.446180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:19.446278 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:19.446370 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:19.447076 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.447768 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.449049 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.449442 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.449575 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:19.453448 (+ 3873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59503: BlockingRecv error: recv got EOF from 127.0.97.2:59503 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 06:55:19 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:19 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:19.462260 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:19.454375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:58385)
0927 06:55:19.454539 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:19.454558 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:19.454648 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:19.454949 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:19.454954 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:19.455243 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:19.455378 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:19.455385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.456554 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.456559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:19.457470 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:19.457480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.458177 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.458182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:19.458352 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:19.458983 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:19.459004 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:19.461952 (+ 2948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 06:55:19.462348 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:19.454480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58385 (local address 127.0.97.62:42163)
0927 06:55:19.454654 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:19.454659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:19.454677 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:55:19.454733 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:19.454738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:19.454802 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:19.454931 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:19.455504 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.456396 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.457609 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.457995 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.458161 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:19.462148 (+ 3987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58385: BlockingRecv error: recv got EOF from 127.0.97.2:58385 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 06:55:19.462512 9498 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 06:55:19.746352 9501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:19.746601 9501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:19.746824 9501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:19.753897 9501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:19.754020 9501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:19.754249 9501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:55:19.763854 9501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33885
--tserver_master_addrs=127.0.97.62:42163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:19.765075 9501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:19.766450 9501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:19.779486 9506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:19.779524 9507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:19.779567 9509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:19.785517 9501 server_base.cc:1034] running on GCE node
I20260927 06:55:19.786165 9501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:19.787012 9501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:19.788199 9501 hybrid_clock.cc:648] HybridClock initialized: now 1790492119788160 us; error 44 us; skew 500 ppm
I20260927 06:55:19.788491 9501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:19 dist-test-slave-f892 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492119, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:19.791889 9501 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 06:55:19.794549 9501 webserver.cc:460] Webserver started at http://127.0.97.3:33885/ using document root <none> and password file <none>
I20260927 06:55:19.795575 9501 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:19.795735 9501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:19.803319 9501 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 06:55:19.807483 9516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:19.809196 9501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 06:55:19.809415 9501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "1f891814adb14482bedeeb5d6de0f5f4"
format_stamp: "Formatted at 2026-09-27 06:55:17 on dist-test-slave-f892"
server_key: "4aa4b11128a0ab77f0488c33429970e9"
server_key_iv: "95a077b175f296423ac5425e246a4612"
server_key_version: "encryptionkey@0"
I20260927 06:55:19.810768 9501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:19.840402 9501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:19.844223 9501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:19.844488 9501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:19.845479 9501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:19.847025 9501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:19.847146 9501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:19.847293 9501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:19.847365 9501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:19.879202 9501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34701
I20260927 06:55:19.879284 9628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34701 every 8 connection(s)
I20260927 06:55:19.880529 9501 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:55:19.885565 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 9501
Sep 27 06:55:19 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:19 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 06:55:19.898453 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:19.883125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45185 (local address 127.0.97.62:42163)
0927 06:55:19.883300 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:19.883305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:19.884447 (+ 1142us) server_negotiation.cc:407] Connection header received
0927 06:55:19.886875 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:19.886883 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:19.886973 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:19.887055 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:19.889401 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.890787 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.892947 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.893506 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.893641 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:19.898178 (+ 4537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45185: BlockingRecv error: recv got EOF from 127.0.97.3:45185 (error 108)
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":60}
I20260927 06:55:19.899036 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:19.883622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.3:45185)
0927 06:55:19.884297 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:19.884332 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:19.886528 (+ 2196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:19.887578 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:19.887590 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:19.888389 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:19.889187 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:19.889214 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.890942 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.890947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:19.892831 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:19.892841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.893733 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.893741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:19.893943 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:19.894597 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:19.894622 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:19.897961 (+ 3339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":281,"threads_started":1}
W20260927 06:55:19.899681 9629 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42163 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:42163: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260927 06:55:19.901924 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:19.886644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.0.97.62:42163)
0927 06:55:19.887312 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:19.887317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:19.887341 (+ 24us) server_negotiation.cc:407] Connection header received
0927 06:55:19.887380 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:19.887383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:19.887432 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:19.887511 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:19.889401 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.890787 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.892587 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.893046 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.893162 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:19.895709 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:19.895717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:19.895732 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:19.895769 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:19.898756 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:19.899445 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:19.899455 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:19.899466 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:19.899606 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:19.899991 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:19.899996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:19.900001 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:19.900573 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:19.900894 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:19.901310 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 06:55:19.901635 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":589,"thread_start_us":460,"threads_started":1}
Sep 27 06:55:19 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:19 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:19.907722 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:19.900200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.3:43017)
0927 06:55:19.900362 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:19.900379 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:19.900469 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:19.900740 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:19.900744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:19.901041 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:19.901168 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:19.901175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.902306 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.902311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:19.903463 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:19.903474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.904332 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.904336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:19.904448 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:19.904938 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:19.904953 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:19.907529 (+ 2576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 06:55:19.907800 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:19.900292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43017 (local address 127.0.97.62:42163)
0927 06:55:19.900417 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:19.900421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:19.900435 (+ 14us) server_negotiation.cc:407] Connection header received
0927 06:55:19.900568 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:19.900572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:19.900617 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:19.900721 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:19.901288 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.902166 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.903589 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.904187 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.904306 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:19.907647 (+ 3341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43017: BlockingRecv error: recv got EOF from 127.0.97.3:43017 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 27 06:55:19 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:19 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:19.914809 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:19.908442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.3:58927)
0927 06:55:19.908597 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:19.908611 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:19.908714 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:19.908930 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:19.908933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:19.909159 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:19.909278 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:19.909284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.910235 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.910240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:19.911111 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:19.911121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.911693 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.911697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:19.911807 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:19.912255 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:19.912269 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:19.914556 (+ 2287us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 06:55:19.915053 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:19.908496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58927 (local address 127.0.97.62:42163)
0927 06:55:19.908700 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:19.908704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:19.908718 (+ 14us) server_negotiation.cc:407] Connection header received
0927 06:55:19.908782 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:19.908785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:19.908828 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:19.908899 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:19.909381 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.910124 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.911237 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:19.911585 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:19.911672 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:19.914761 (+ 3089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58927: BlockingRecv error: recv got EOF from 127.0.97.3:58927 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260927 06:55:19.915066 9629 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 06:55:20 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:20 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:20.015607 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:20.006944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:51875)
0927 06:55:20.007464 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:20.007479 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:20.007599 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:20.007893 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:20.007896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:20.008358 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:20.008612 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:20.008621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:20.010093 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:20.010097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:20.011176 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:20.011187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:20.011850 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:20.011854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:20.011968 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:20.012525 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:20.012541 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:20.015285 (+ 2744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":263,"threads_started":1}
W20260927 06:55:20.015678 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:20.006999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51875 (local address 127.0.97.62:42163)
0927 06:55:20.007208 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:20.007213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:20.007572 (+ 359us) server_negotiation.cc:407] Connection header received
0927 06:55:20.007695 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:20.007700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:20.007768 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:20.007850 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:20.008771 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:20.009865 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:20.011304 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:20.011695 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:20.011798 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:20.015465 (+ 3667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51875: BlockingRecv error: recv got EOF from 127.0.97.1:51875 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 27 06:55:20 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:20 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:20.471621 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:20.463051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:38405)
0927 06:55:20.463564 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:20.463579 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:20.463684 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:20.463993 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:20.463996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:20.464414 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:20.464592 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:20.464600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:20.465536 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:20.465539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:20.466746 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:20.466762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:20.467683 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:20.467689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:20.467850 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:20.468528 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:20.468543 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:20.471404 (+ 2861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":303,"threads_started":1}
W20260927 06:55:20.471819 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:20.463143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38405 (local address 127.0.97.62:42163)
0927 06:55:20.463316 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:20.463320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:20.463651 (+ 331us) server_negotiation.cc:407] Connection header received
0927 06:55:20.463775 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:20.463779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:20.463863 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:20.463952 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:20.464712 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:20.465423 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:20.466899 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:20.467533 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:20.467650 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:20.471572 (+ 3922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38405: BlockingRecv error: recv got EOF from 127.0.97.2:38405 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 06:55:20 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:20 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:20.923959 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:20.915686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.3:59759)
0927 06:55:20.916163 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:20.916178 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:20.916284 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:20.916710 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:20.916713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:20.917205 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:20.917439 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:20.917448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:20.918535 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:20.918542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:20.919480 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:20.919491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:20.920204 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:20.920210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:20.920380 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:20.920922 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:20.920939 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:20.923630 (+ 2691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":250,"threads_started":1}
W20260927 06:55:20.924043 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:20.915739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59759 (local address 127.0.97.62:42163)
0927 06:55:20.915891 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:20.915895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:20.916244 (+ 349us) server_negotiation.cc:407] Connection header received
0927 06:55:20.916385 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:20.916389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:20.916448 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:20.916530 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:20.917582 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:20.918390 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:20.919625 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:20.920027 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:20.920161 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:20.923816 (+ 3655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59759: BlockingRecv error: recv got EOF from 127.0.97.3:59759 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 06:55:21 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:21 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:21.024801 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:21.016538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:45221)
0927 06:55:21.017118 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:21.017135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:21.017251 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:21.017532 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:21.017535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:21.017926 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:21.018099 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:21.018107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:21.019103 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:21.019107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:21.019995 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:21.020006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:21.020696 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:21.020700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:21.020807 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:21.021412 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:21.021428 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:21.024468 (+ 3040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":350,"threads_started":1}
W20260927 06:55:21.024909 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:21.016638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45221 (local address 127.0.97.62:42163)
0927 06:55:21.016800 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:21.016804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:21.017205 (+ 401us) server_negotiation.cc:407] Connection header received
0927 06:55:21.017350 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:21.017354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:21.017412 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:21.017491 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:21.018241 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:21.018994 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:21.020140 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:21.020566 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:21.020667 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:21.024665 (+ 3998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45221: BlockingRecv error: recv got EOF from 127.0.97.1:45221 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 06:55:21 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:21 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:21.480345 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:21.472499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:37085)
0927 06:55:21.472976 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:21.472992 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:21.473102 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:21.473399 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:21.473403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:21.473784 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:21.473919 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:21.473926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:21.474844 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:21.474848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:21.475733 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:21.475743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:21.476375 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:21.476379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:21.476477 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:21.477121 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:21.477137 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:21.479984 (+ 2847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":289,"threads_started":1}
W20260927 06:55:21.480391 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:21.472547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37085 (local address 127.0.97.62:42163)
0927 06:55:21.472779 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:21.472788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:21.473068 (+ 280us) server_negotiation.cc:407] Connection header received
0927 06:55:21.473212 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:21.473216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:21.473271 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:21.473350 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:21.474078 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:21.474738 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:21.475872 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:21.476264 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:21.476345 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:21.480186 (+ 3841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37085: BlockingRecv error: recv got EOF from 127.0.97.2:37085 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 06:55:21 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:21 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:21.932497 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:21.924958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.3:41317)
0927 06:55:21.925458 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:21.925473 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:21.925581 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:21.925887 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:21.925890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:21.926284 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:21.926431 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:21.926439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:21.927383 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:21.927386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:21.928281 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:21.928291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:21.928930 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:21.928934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:21.929040 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:21.929652 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:21.929668 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:21.932269 (+ 2601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":291,"threads_started":1}
W20260927 06:55:21.932693 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:21.925122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41317 (local address 127.0.97.62:42163)
0927 06:55:21.925343 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:21.925349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:21.925557 (+ 208us) server_negotiation.cc:407] Connection header received
0927 06:55:21.925674 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:21.925678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:21.925753 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:21.925833 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:21.926559 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:21.927265 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:21.928396 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:21.928807 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:21.928899 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:21.932436 (+ 3537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41317: BlockingRecv error: recv got EOF from 127.0.97.3:41317 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 06:55:22 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:22 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:22.033792 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:22.025934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:38671)
0927 06:55:22.026455 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:22.026472 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:22.026593 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:22.026886 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:22.026890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:22.027212 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:22.027389 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:22.027397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:22.028434 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:22.028442 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:22.029338 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:22.029349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:22.030062 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:22.030068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:22.030243 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:22.030852 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:22.030869 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:22.033465 (+ 2596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":333,"threads_started":1}
W20260927 06:55:22.033874 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:22.025989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38671 (local address 127.0.97.62:42163)
0927 06:55:22.026371 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:22.026377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:22.026557 (+ 180us) server_negotiation.cc:407] Connection header received
0927 06:55:22.026698 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:22.026703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:22.026763 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:22.026843 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:22.027544 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:22.028298 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:22.029479 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:22.029877 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:22.030000 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:22.033643 (+ 3643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38671: BlockingRecv error: recv got EOF from 127.0.97.1:38671 (error 108)
Metrics: {"server-negotiator.queue_time_us":216}
Sep 27 06:55:22 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:22 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:22.489086 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:22.481181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:54497)
0927 06:55:22.481737 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:22.481752 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:22.481863 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:22.482145 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:22.482148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:22.482513 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:22.482668 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:22.482676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:22.483658 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:22.483661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:22.484516 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:22.484527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:22.485237 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:22.485245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:22.485420 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:22.486073 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:22.486091 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:22.488709 (+ 2618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":304,"threads_started":1}
W20260927 06:55:22.489145 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:22.481272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54497 (local address 127.0.97.62:42163)
0927 06:55:22.481429 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:22.481433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:22.481820 (+ 387us) server_negotiation.cc:407] Connection header received
0927 06:55:22.481952 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:22.481955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:22.482058 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:22.482132 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:22.482790 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:22.483541 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:22.484652 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:22.485087 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:22.485257 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:22.488886 (+ 3629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54497: BlockingRecv error: recv got EOF from 127.0.97.2:54497 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 06:55:22 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:22 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:22.940989 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:22.933338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.3:47969)
0927 06:55:22.933803 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:22.933818 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:22.933964 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:22.934346 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:22.934352 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:22.934687 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:22.934832 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:22.934841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:22.935800 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:22.935810 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:22.936695 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:22.936705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:22.937354 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:22.937360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:22.937533 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:22.938200 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:22.938216 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:22.940759 (+ 2543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":243,"threads_started":1}
W20260927 06:55:22.941141 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:22.933416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47969 (local address 127.0.97.62:42163)
0927 06:55:22.933639 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:22.933647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:22.933895 (+ 248us) server_negotiation.cc:407] Connection header received
0927 06:55:22.934121 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:22.934128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:22.934202 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:22.934305 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:22.934963 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:22.935642 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:22.936833 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:22.937221 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:22.937337 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:22.940917 (+ 3580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47969: BlockingRecv error: recv got EOF from 127.0.97.3:47969 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 06:55:23 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:23 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:23.043049 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:23.035008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:58133)
0927 06:55:23.035754 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:23.035771 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:23.035953 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:23.036297 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:23.036301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:23.036661 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:23.036807 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:23.036814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:23.037773 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:23.037781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:23.038757 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:23.038767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:23.039474 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:23.039481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:23.039668 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:23.040208 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:23.040240 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:23.042838 (+ 2598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":456,"threads_started":1}
W20260927 06:55:23.043233 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:23.035116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58133 (local address 127.0.97.62:42163)
0927 06:55:23.035308 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:23.035316 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:23.035890 (+ 574us) server_negotiation.cc:407] Connection header received
0927 06:55:23.036081 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:23.036087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:23.036150 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:23.036244 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:23.036941 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:23.037633 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:23.038916 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:23.039336 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:23.039464 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:23.043020 (+ 3556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58133: BlockingRecv error: recv got EOF from 127.0.97.1:58133 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 06:55:23 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:23 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:23.498173 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:23.490042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:42397)
0927 06:55:23.490496 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:23.490510 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:23.490665 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:23.491018 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:23.491023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:23.491394 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:23.491542 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:23.491550 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:23.492542 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:23.492550 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:23.493517 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:23.493528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:23.494259 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:23.494264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:23.494448 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:23.495010 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:23.495026 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:23.497832 (+ 2806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":251,"threads_started":1}
W20260927 06:55:23.498266 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:23.490100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42397 (local address 127.0.97.62:42163)
0927 06:55:23.490314 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:23.490320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:23.490593 (+ 273us) server_negotiation.cc:407] Connection header received
0927 06:55:23.490789 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:23.490795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:23.490883 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:23.490985 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:23.491683 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:23.492414 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:23.493653 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:23.494127 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:23.494250 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:23.498045 (+ 3795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42397: BlockingRecv error: recv got EOF from 127.0.97.2:42397 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 06:55:23 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:23 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:23.949438 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:23.941986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.3:40055)
0927 06:55:23.942569 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:23.942584 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:23.942686 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:23.943021 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:23.943027 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:23.943388 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:23.943526 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:23.943533 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:23.944450 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:23.944453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:23.945372 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:23.945381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:23.945980 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:23.945984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:23.946129 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:23.946652 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:23.946668 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:23.949230 (+ 2562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":316,"threads_started":1}
W20260927 06:55:23.949620 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:23.942039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40055 (local address 127.0.97.62:42163)
0927 06:55:23.942219 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:23.942225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:23.942644 (+ 419us) server_negotiation.cc:407] Connection header received
0927 06:55:23.942774 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:23.942780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:23.942862 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:23.942974 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:23.943638 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:23.944343 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:23.945475 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:23.945871 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:23.945954 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:23.949407 (+ 3453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40055: BlockingRecv error: recv got EOF from 127.0.97.3:40055 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 06:55:24 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:24 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:24.052346 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:24.044133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:60665)
0927 06:55:24.044639 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:24.044656 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:24.044762 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:24.045041 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:24.045044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:24.045440 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:24.045594 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:24.045602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:24.046614 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:24.046618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:24.047692 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:24.047705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:24.048506 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:24.048512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:24.048677 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:24.049283 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:24.049299 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:24.051973 (+ 2674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":259,"threads_started":1}
W20260927 06:55:24.052387 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:24.044260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60665 (local address 127.0.97.62:42163)
0927 06:55:24.044455 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:24.044459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:24.044729 (+ 270us) server_negotiation.cc:407] Connection header received
0927 06:55:24.044861 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:24.044865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:24.044920 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:24.044998 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:24.045714 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:24.046485 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:24.047849 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:24.048340 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:24.048478 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:24.052196 (+ 3718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60665: BlockingRecv error: recv got EOF from 127.0.97.1:60665 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 27 06:55:24 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:24 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:24.510401 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:24.499087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:60571)
0927 06:55:24.499667 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:24.499680 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:24.499821 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:24.500134 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:24.500138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:24.500544 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:24.500696 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:24.500704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:24.501685 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:24.501692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:24.502669 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:24.502680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:24.503382 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:24.503388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:24.503552 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:24.507064 (+ 3512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:24.507080 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:24.510060 (+ 2980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":260,"threads_started":1}
W20260927 06:55:24.510490 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:24.499268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60571 (local address 127.0.97.62:42163)
0927 06:55:24.499477 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:24.499484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:24.499756 (+ 272us) server_negotiation.cc:407] Connection header received
0927 06:55:24.499944 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:24.499954 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:24.500011 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:24.500106 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:24.500843 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:24.501547 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:24.502806 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:24.503230 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:24.503372 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:24.510235 (+ 6863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60571: BlockingRecv error: recv got EOF from 127.0.97.2:60571 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 06:55:24 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:24 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:24.958060 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:24.950319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.3:46069)
0927 06:55:24.950834 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:24.950849 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:24.950948 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:24.951221 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:24.951224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:24.951619 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:24.951786 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:24.951795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:24.952745 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:24.952748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:24.953628 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:24.953637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:24.954201 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:24.954204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:24.954317 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:24.954829 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:24.954844 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:24.957672 (+ 2828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":266,"threads_started":1}
W20260927 06:55:24.958302 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:24.950416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46069 (local address 127.0.97.62:42163)
0927 06:55:24.950597 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:24.950601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:24.950921 (+ 320us) server_negotiation.cc:407] Connection header received
0927 06:55:24.951042 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:24.951046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:24.951101 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:24.951180 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:24.951902 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:24.952632 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:24.953733 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:24.954099 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:24.954179 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:24.957987 (+ 3808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46069: BlockingRecv error: recv got EOF from 127.0.97.3:46069 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 06:55:25 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:25 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:25.061245 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:25.053329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:48825)
0927 06:55:25.053804 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:25.053829 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:25.053979 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:25.054322 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:25.054327 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:25.054705 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:25.054851 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:25.054858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:25.055840 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:25.055851 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:25.056735 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:25.056745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:25.057439 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:25.057445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:25.057619 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:25.058278 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:25.058295 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:25.061038 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":267,"threads_started":1}
W20260927 06:55:25.061446 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:25.053385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48825 (local address 127.0.97.62:42163)
0927 06:55:25.053604 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:25.053612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:25.053914 (+ 302us) server_negotiation.cc:407] Connection header received
0927 06:55:25.054126 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:25.054134 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:25.054198 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:25.054294 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:25.054983 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:25.055701 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:25.056877 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:25.057286 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:25.057424 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:25.061228 (+ 3804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48825: BlockingRecv error: recv got EOF from 127.0.97.1:48825 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 06:55:25 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:25 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:25.520216 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:25.511368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:46539)
0927 06:55:25.511914 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:25.511929 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:25.512035 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:25.512342 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:25.512345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:25.512689 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:25.512862 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:25.512870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:25.514249 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:25.514253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:25.515313 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:25.515325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:25.516249 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:25.516253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:25.516374 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:25.516992 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:25.517009 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:25.519996 (+ 2987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":322,"threads_started":1}
W20260927 06:55:25.520385 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:25.511395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46539 (local address 127.0.97.62:42163)
0927 06:55:25.511574 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:25.511579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:25.512013 (+ 434us) server_negotiation.cc:407] Connection header received
0927 06:55:25.512135 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:25.512141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:25.512213 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:25.512311 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:25.513007 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:25.514135 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:25.515444 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:25.516113 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:25.516223 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:25.520184 (+ 3961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46539: BlockingRecv error: recv got EOF from 127.0.97.2:46539 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 06:55:25 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:25 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:25.966856 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:25.958894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.3:57809)
0927 06:55:25.959397 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:25.959412 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:25.959574 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:25.960008 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:25.960011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:25.960350 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:25.960484 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:25.960491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:25.961472 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:25.961476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:25.962437 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:25.962447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:25.963087 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:25.963091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:25.963206 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:25.963754 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:25.963770 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:25.966642 (+ 2872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":275,"threads_started":1}
W20260927 06:55:25.967023 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:25.958895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57809 (local address 127.0.97.62:42163)
0927 06:55:25.959125 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:25.959134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:25.959504 (+ 370us) server_negotiation.cc:407] Connection header received
0927 06:55:25.959734 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:25.959739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:25.959806 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:25.959857 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:25.960628 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:25.961358 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:25.962553 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:25.962967 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:25.963058 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:25.966817 (+ 3759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57809: BlockingRecv error: recv got EOF from 127.0.97.3:57809 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 06:55:26 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:26 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 06:55:26.070163 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:26.062192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42205 (local address 127.0.97.62:42163)
0927 06:55:26.062420 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:26.062428 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:26.062765 (+ 337us) server_negotiation.cc:407] Connection header received
0927 06:55:26.062951 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:26.062957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:26.063023 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:26.063123 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:26.063858 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:26.064546 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:26.065690 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:26.066174 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:26.066307 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:26.069945 (+ 3638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42205: BlockingRecv error: recv got EOF from 127.0.97.1:42205 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 06:55:26.070168 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:26.062124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:42205)
0927 06:55:26.062650 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:26.062667 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:26.062831 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:26.063151 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:26.063155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:26.063559 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:26.063722 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:26.063730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:26.064677 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:26.064685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:26.065557 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:26.065568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:26.066322 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:26.066327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:26.066490 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:26.067183 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:26.067199 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:26.069783 (+ 2584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":208,"threads_started":1}
Sep 27 06:55:26 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:26 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:26.529275 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:26.521160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:38651)
0927 06:55:26.521755 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:26.521770 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:26.521932 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:26.522303 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:26.522308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:26.522709 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:26.522863 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:26.522878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:26.523957 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:26.523964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:26.524879 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:26.524890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:26.525577 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:26.525583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:26.525749 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:26.526349 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:26.526368 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:26.529035 (+ 2667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":270,"threads_started":1}
W20260927 06:55:26.529459 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:26.521256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38651 (local address 127.0.97.62:42163)
0927 06:55:26.521488 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:26.521495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:26.521851 (+ 356us) server_negotiation.cc:407] Connection header received
0927 06:55:26.522096 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:26.522100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:26.522165 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:26.522268 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:26.523007 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:26.523824 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:26.525014 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:26.525428 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:26.525561 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:26.529217 (+ 3656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38651: BlockingRecv error: recv got EOF from 127.0.97.2:38651 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 06:55:26 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:26 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:26.975693 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:26.967681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.3:44707)
0927 06:55:26.968256 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:26.968272 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:26.968401 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:26.968739 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:26.968742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:26.969214 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:26.969387 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:26.969396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:26.970830 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:26.970833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:26.971723 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:26.971732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:26.972380 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:26.972384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:26.972492 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:26.973012 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:26.973028 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:26.975468 (+ 2440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":265,"threads_started":1}
W20260927 06:55:26.975839 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:26.967756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44707 (local address 127.0.97.62:42163)
0927 06:55:26.967910 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:26.967914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:26.968360 (+ 446us) server_negotiation.cc:407] Connection header received
0927 06:55:26.968505 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:26.968511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:26.968583 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:26.968715 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:26.969525 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:26.970708 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:26.971842 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:26.972257 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:26.972350 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:26.975646 (+ 3296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44707: BlockingRecv error: recv got EOF from 127.0.97.3:44707 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 06:55:27 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:27 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:27.079340 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:27.071084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:54535)
0927 06:55:27.071940 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:27.071959 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:27.072105 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:27.072461 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:27.072467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:27.072870 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:27.073011 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:27.073019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:27.074011 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:27.074017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:27.074977 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:27.074989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:27.075798 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:27.075804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:27.075981 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:27.076554 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:27.076578 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:27.079126 (+ 2548us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":564,"threads_started":1}
W20260927 06:55:27.079527 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:27.071281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54535 (local address 127.0.97.62:42163)
0927 06:55:27.071503 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:27.071509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:27.072047 (+ 538us) server_negotiation.cc:407] Connection header received
0927 06:55:27.072234 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:27.072240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:27.072320 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:27.072430 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:27.073154 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:27.073858 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:27.075149 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:27.075624 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:27.075762 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:27.079315 (+ 3553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54535: BlockingRecv error: recv got EOF from 127.0.97.1:54535 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 27 06:55:27 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:27 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:27.538527 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:27.530283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:56947)
0927 06:55:27.530864 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:27.530883 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:27.531036 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:27.531406 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:27.531411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:27.531927 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:27.532128 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:27.532138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:27.533379 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:27.533383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:27.534303 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:27.534314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:27.534996 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:27.535000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:27.535110 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:27.535647 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:27.535663 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:27.538185 (+ 2522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":342,"threads_started":1}
W20260927 06:55:27.538532 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:27.530413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56947 (local address 127.0.97.62:42163)
0927 06:55:27.530612 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:27.530620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:27.531010 (+ 390us) server_negotiation.cc:407] Connection header received
0927 06:55:27.531153 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:27.531160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:27.531240 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:27.531348 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:27.532283 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:27.533248 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:27.534417 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:27.534845 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:27.534949 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:27.538368 (+ 3419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56947: BlockingRecv error: recv got EOF from 127.0.97.2:56947 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 06:55:27 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:27 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:27.984948 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:27.976666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.3:41057)
0927 06:55:27.977132 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:27.977147 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:27.977308 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:27.977656 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:27.977661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:27.978154 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:27.978308 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:27.978316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:27.979307 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:27.979313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:27.980192 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:27.980203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:27.981121 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:27.981128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:27.981306 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:27.982056 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:27.982079 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:27.984742 (+ 2663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":260,"threads_started":1}
W20260927 06:55:27.985227 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:27.976717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41057 (local address 127.0.97.62:42163)
0927 06:55:27.976987 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:27.976993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:27.977231 (+ 238us) server_negotiation.cc:407] Connection header received
0927 06:55:27.977430 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:27.977436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:27.977511 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:27.977628 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:27.978442 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:27.979158 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:27.980337 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:27.980945 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:27.981069 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:27.984904 (+ 3835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41057: BlockingRecv error: recv got EOF from 127.0.97.3:41057 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 27 06:55:28 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:28 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:28.088554 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:28.080189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:53523)
0927 06:55:28.080689 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:28.080705 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:28.080872 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:28.081210 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:28.081214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:28.081637 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:28.081791 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:28.081801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:28.082950 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:28.082957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:28.083883 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:28.083893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:28.084574 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:28.084579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:28.084742 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:28.085275 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:28.085301 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:28.088186 (+ 2885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":273,"threads_started":1}
W20260927 06:55:28.088822 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:28.080269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53523 (local address 127.0.97.62:42163)
0927 06:55:28.080509 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:28.080516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:28.080802 (+ 286us) server_negotiation.cc:407] Connection header received
0927 06:55:28.081003 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:28.081008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:28.081063 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:28.081160 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:28.081922 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:28.082834 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:28.084015 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:28.084429 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:28.084563 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:28.088396 (+ 3833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53523: BlockingRecv error: recv got EOF from 127.0.97.1:53523 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 06:55:28 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:28 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:28.547024 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:28.539453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:41931)
0927 06:55:28.539909 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:28.539925 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:28.540032 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:28.540380 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:28.540383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:28.540726 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:28.540887 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:28.540895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:28.541801 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:28.541805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:28.542802 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:28.542814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:28.543427 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:28.543431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:28.543547 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:28.544077 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:28.544093 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:28.546712 (+ 2619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":244,"threads_started":1}
W20260927 06:55:28.547202 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:28.539460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41931 (local address 127.0.97.62:42163)
0927 06:55:28.539655 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:28.539665 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:28.539998 (+ 333us) server_negotiation.cc:407] Connection header received
0927 06:55:28.540150 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:28.540158 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:28.540241 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:28.540344 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:28.541001 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:28.541692 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:28.542917 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:28.543309 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:28.543398 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:28.546960 (+ 3562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41931: BlockingRecv error: recv got EOF from 127.0.97.2:41931 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 06:55:28 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:28 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:28.993636 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:28.985829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.3:42907)
0927 06:55:28.986357 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:28.986373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:28.986480 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:28.986732 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:28.986735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:28.987063 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:28.987235 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:28.987243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:28.988171 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:28.988174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:28.989046 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:28.989055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:28.989841 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:28.989845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:28.989971 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:28.990643 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:28.990659 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:28.993414 (+ 2755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":294,"threads_started":1}
W20260927 06:55:28.993764 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:28.985867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42907 (local address 127.0.97.62:42163)
0927 06:55:28.986050 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:28.986058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:28.986442 (+ 384us) server_negotiation.cc:407] Connection header received
0927 06:55:28.986559 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:28.986562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:28.986616 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:28.986691 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:28.987355 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:28.988074 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:28.989157 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:28.989707 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:28.989814 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:28.993587 (+ 3773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42907: BlockingRecv error: recv got EOF from 127.0.97.3:42907 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 06:55:29 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:29 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:29.097292 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:29.089478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:43529)
0927 06:55:29.089940 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:29.089956 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:29.090123 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:29.090433 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:29.090437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:29.090826 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:29.090976 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:29.090984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:29.091988 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:29.091995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:29.092914 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:29.092924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:29.093661 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:29.093666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:29.093835 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:29.094395 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:29.094410 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:29.097044 (+ 2634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":256,"threads_started":1}
W20260927 06:55:29.097644 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:29.089542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43529 (local address 127.0.97.62:42163)
0927 06:55:29.089762 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:29.089770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:29.090043 (+ 273us) server_negotiation.cc:407] Connection header received
0927 06:55:29.090244 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:29.090250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:29.090317 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:29.090409 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:29.091118 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:29.091858 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:29.093051 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:29.093495 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:29.093631 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:29.097246 (+ 3615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43529: BlockingRecv error: recv got EOF from 127.0.97.1:43529 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 06:55:29 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:29 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:29.560107 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:29.547983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:34819)
0927 06:55:29.548405 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:29.548421 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:29.548525 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:29.548805 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:29.548808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:29.549147 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:29.549285 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:29.549293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:29.550312 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:29.550316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:29.551229 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:29.551240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:29.551853 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:29.551857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:29.551970 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:29.557145 (+ 5175us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:29.557162 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:29.559896 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":241,"threads_started":1}
W20260927 06:55:29.560307 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:29.548027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34819 (local address 127.0.97.62:42163)
0927 06:55:29.548240 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:29.548247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:29.548502 (+ 255us) server_negotiation.cc:407] Connection header received
0927 06:55:29.548621 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:29.548625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:29.548682 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:29.548767 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:29.549406 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:29.550187 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:29.551342 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:29.551717 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:29.551814 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:29.560056 (+ 8242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34819: BlockingRecv error: recv got EOF from 127.0.97.2:34819 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 06:55:29 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:29 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:30.002157 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:29.994559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.3:38051)
0927 06:55:29.995084 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:29.995101 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:29.995203 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:29.995471 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:29.995474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:29.995812 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:29.995959 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:29.995967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:29.996874 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:29.996878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:29.997832 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:29.997842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:29.998557 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:29.998561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:29.998685 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:29.999206 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:29.999221 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:30.001798 (+ 2577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":298,"threads_started":1}
W20260927 06:55:30.002311 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:29.994692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38051 (local address 127.0.97.62:42163)
0927 06:55:29.994889 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:29.994893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:29.995173 (+ 280us) server_negotiation.cc:407] Connection header received
0927 06:55:29.995295 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:29.995299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:29.995354 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:29.995430 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:29.996081 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:29.996756 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:29.997952 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:29.998419 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:29.998528 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:30.001980 (+ 3452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38051: BlockingRecv error: recv got EOF from 127.0.97.3:38051 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 27 06:55:30 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:30 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:30.106334 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:30.098161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:38185)
0927 06:55:30.098673 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:30.098689 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:30.098849 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:30.099201 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:30.099206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:30.099644 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:30.099820 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:30.099848 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:30.100833 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:30.100839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:30.101741 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:30.101752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:30.102557 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:30.102563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:30.102747 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:30.103338 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:30.103356 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:30.106086 (+ 2730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":266,"threads_started":1}
W20260927 06:55:30.106552 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:30.098179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38185 (local address 127.0.97.62:42163)
0927 06:55:30.098479 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:30.098487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:30.098782 (+ 295us) server_negotiation.cc:407] Connection header received
0927 06:55:30.098986 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:30.098994 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:30.099057 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:30.099170 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:30.099987 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:30.100701 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:30.101882 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:30.102406 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:30.102545 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:30.106276 (+ 3731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38185: BlockingRecv error: recv got EOF from 127.0.97.1:38185 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 06:55:30 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:30 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:30.568457 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:30.560908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:43721)
0927 06:55:30.561400 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:30.561416 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:30.561525 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:30.561808 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:30.561811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:30.562215 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:30.562394 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:30.562402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:30.563369 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:30.563372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:30.564228 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:30.564238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:30.564859 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:30.564863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:30.564968 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:30.565512 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:30.565528 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:30.568161 (+ 2633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":300,"threads_started":1}
W20260927 06:55:30.568670 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:30.561007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43721 (local address 127.0.97.62:42163)
0927 06:55:30.561275 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:30.561281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:30.561494 (+ 213us) server_negotiation.cc:407] Connection header received
0927 06:55:30.561631 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:30.561634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:30.561694 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:30.561771 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:30.562522 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:30.563259 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:30.564342 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:30.564736 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:30.564839 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:30.568399 (+ 3560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43721: BlockingRecv error: recv got EOF from 127.0.97.2:43721 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 06:55:31 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:31 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:31.011685 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:31.003223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.3:47571)
0927 06:55:31.003917 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:31.003937 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:31.004074 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:31.004459 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:31.004462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:31.004838 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:31.005002 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:31.005011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:31.005990 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:31.005994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:31.007021 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:31.007032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:31.007740 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:31.007745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:31.007862 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:31.008579 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:31.008609 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:31.011420 (+ 2811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":430,"threads_started":1}
W20260927 06:55:31.011862 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:31.003245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47571 (local address 127.0.97.62:42163)
0927 06:55:31.003436 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:31.003445 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:31.004022 (+ 577us) server_negotiation.cc:407] Connection header received
0927 06:55:31.004247 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:31.004254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:31.004330 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:31.004403 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:31.005179 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:31.005880 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:31.007143 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:31.007610 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:31.007706 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:31.011649 (+ 3943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47571: BlockingRecv error: recv got EOF from 127.0.97.3:47571 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 06:55:31 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:31 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:31.115593 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:31.107252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:54931)
0927 06:55:31.107769 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:31.107785 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:31.107903 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:31.108173 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:31.108176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:31.108581 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:31.108825 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:31.108835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:31.110244 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:31.110249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:31.111128 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:31.111139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:31.111857 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:31.111861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:31.111990 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:31.112553 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:31.112570 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:31.115374 (+ 2804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":269,"threads_started":1}
W20260927 06:55:31.115924 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:31.107298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54931 (local address 127.0.97.62:42163)
0927 06:55:31.107473 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:31.107477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:31.107880 (+ 403us) server_negotiation.cc:407] Connection header received
0927 06:55:31.107994 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:31.107997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:31.108055 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:31.108141 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:31.108974 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:31.110117 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:31.111277 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:31.111735 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:31.111840 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:31.115558 (+ 3718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54931: BlockingRecv error: recv got EOF from 127.0.97.1:54931 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 06:55:31 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:31 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:31.577304 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:31.569503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:38715)
0927 06:55:31.569980 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:31.569995 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:31.570206 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:31.570525 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:31.570528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:31.570880 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:31.571030 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:31.571038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:31.572119 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:31.572122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:31.573022 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:31.573033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:31.573673 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:31.573677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:31.573804 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:31.574387 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:31.574404 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:31.576986 (+ 2582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":265,"threads_started":1}
W20260927 06:55:31.577445 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:31.569521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38715 (local address 127.0.97.62:42163)
0927 06:55:31.569745 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:31.569754 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:31.570141 (+ 387us) server_negotiation.cc:407] Connection header received
0927 06:55:31.570330 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:31.570333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:31.570425 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:31.570503 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:31.571165 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:31.571984 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:31.573147 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:31.573550 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:31.573652 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:31.577228 (+ 3576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38715: BlockingRecv error: recv got EOF from 127.0.97.2:38715 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 06:55:32 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:32 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:32.021270 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:32.012706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.3:44231)
0927 06:55:32.013212 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:32.013228 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:32.013347 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:32.013660 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:32.013665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:32.014178 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:32.014441 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:32.014454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:32.015758 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:32.015761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:32.016768 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:32.016779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:32.017440 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:32.017445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:32.017578 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:32.018229 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:32.018246 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:32.020944 (+ 2698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":247,"threads_started":1}
W20260927 06:55:32.021401 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:32.012712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44231 (local address 127.0.97.62:42163)
0927 06:55:32.012926 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:32.012936 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:32.013312 (+ 376us) server_negotiation.cc:407] Connection header received
0927 06:55:32.013449 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:32.013453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:32.013513 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:32.013593 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:32.014589 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:32.015633 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:32.016905 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:32.017320 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:32.017420 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:32.021163 (+ 3743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44231: BlockingRecv error: recv got EOF from 127.0.97.3:44231 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 06:55:32 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:32 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:32.124949 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:32.116480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:45223)
0927 06:55:32.117102 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:32.117117 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:32.117276 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:32.117596 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:32.117600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:32.118026 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:32.118206 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:32.118216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:32.119249 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:32.119256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:32.120281 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:32.120293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:32.121169 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:32.121186 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:32.121341 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:32.121964 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:32.121982 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:32.124604 (+ 2622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":353,"threads_started":1}
W20260927 06:55:32.125020 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:32.116556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45223 (local address 127.0.97.62:42163)
0927 06:55:32.116929 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:32.116937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:32.117206 (+ 269us) server_negotiation.cc:407] Connection header received
0927 06:55:32.117400 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:32.117411 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:32.117468 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:32.117570 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:32.118351 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:32.119100 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:32.120420 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:32.120975 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:32.121111 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:32.124788 (+ 3677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45223: BlockingRecv error: recv got EOF from 127.0.97.1:45223 (error 108)
Metrics: {"server-negotiator.queue_time_us":197}
Sep 27 06:55:32 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:32 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 06:55:32.588574 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:32.578466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41329 (local address 127.0.97.62:42163)
0927 06:55:32.578624 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:32.578628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:32.579012 (+ 384us) server_negotiation.cc:407] Connection header received
0927 06:55:32.579182 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:32.579186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:32.579254 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:32.579330 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:32.580327 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:32.581074 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:32.583213 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:32.583604 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:32.583695 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:32.588408 (+ 4713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41329: BlockingRecv error: recv got EOF from 127.0.97.2:41329 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 06:55:32.588586 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:32.578335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:41329)
0927 06:55:32.578903 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:32.578927 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:32.579068 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:32.579392 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:32.579397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:32.579938 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:32.580184 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:32.580198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:32.581252 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:32.581261 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:32.583062 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:32.583081 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:32.583791 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:32.583798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:32.584009 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:32.584619 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:32.584644 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:32.588181 (+ 3537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":273,"threads_started":1}
Sep 27 06:55:33 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:33 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:33.029742 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:33.022275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.3:52699)
0927 06:55:33.022844 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:33.022884 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:33.023019 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:33.023358 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:33.023361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:33.023725 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:33.023873 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:33.023881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:33.024798 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:33.024801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:33.025701 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:33.025712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:33.026375 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:33.026380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:33.026489 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:33.027029 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:33.027045 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:33.029522 (+ 2477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":301,"threads_started":1}
W20260927 06:55:33.029913 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:33.022280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52699 (local address 127.0.97.62:42163)
0927 06:55:33.022500 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:33.022510 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:33.022973 (+ 463us) server_negotiation.cc:407] Connection header received
0927 06:55:33.023171 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:33.023176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:33.023238 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:33.023318 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:33.023988 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:33.024671 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:33.025821 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:33.026232 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:33.026320 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:33.029717 (+ 3397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52699: BlockingRecv error: recv got EOF from 127.0.97.3:52699 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 06:55:33 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:33 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:33.134495 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:33.126143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:58297)
0927 06:55:33.126642 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:33.126659 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:33.126766 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:33.127093 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:33.127097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:33.127444 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:33.127676 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:33.127691 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:33.129137 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:33.129142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:33.130063 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:33.130074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:33.130702 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:33.130706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:33.130817 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:33.131474 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:33.131491 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:33.134252 (+ 2761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":259,"threads_started":1}
W20260927 06:55:33.134670 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:33.126227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58297 (local address 127.0.97.62:42163)
0927 06:55:33.126487 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:33.126491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:33.126734 (+ 243us) server_negotiation.cc:407] Connection header received
0927 06:55:33.126901 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:33.126906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:33.126968 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:33.127056 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:33.127832 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:33.128992 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:33.130194 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:33.130585 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:33.130679 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:33.134451 (+ 3772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58297: BlockingRecv error: recv got EOF from 127.0.97.1:58297 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 27 06:55:33 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:33 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:33.597586 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:33.589636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:48997)
0927 06:55:33.590126 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:33.590142 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:33.590253 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:33.590509 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:33.590512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:33.590854 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:33.590999 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:33.591007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:33.591958 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:33.591962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:33.592855 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:33.592866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:33.593853 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:33.593885 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:33.594103 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:33.594672 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:33.594689 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:33.597290 (+ 2601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":229,"threads_started":1}
W20260927 06:55:33.597692 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:33.589711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48997 (local address 127.0.97.62:42163)
0927 06:55:33.589915 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:33.589921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:33.590198 (+ 277us) server_negotiation.cc:407] Connection header received
0927 06:55:33.590332 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:33.590335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:33.590392 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:33.590468 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:33.591114 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:33.591844 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:33.592978 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:33.593706 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:33.593821 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:33.597488 (+ 3667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48997: BlockingRecv error: recv got EOF from 127.0.97.2:48997 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 06:55:34 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:34 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:34.038982 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:34.030740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.3:41711)
0927 06:55:34.031343 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:34.031358 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:34.031514 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:34.031860 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:34.031865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:34.032282 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:34.032427 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:34.032436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:34.033544 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:34.033551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:34.034519 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:34.034531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:34.035251 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:34.035257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:34.035416 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:34.035964 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:34.035980 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:34.038635 (+ 2655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":380,"threads_started":1}
W20260927 06:55:34.039083 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:34.030871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41711 (local address 127.0.97.62:42163)
0927 06:55:34.031191 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:34.031197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:34.031438 (+ 241us) server_negotiation.cc:407] Connection header received
0927 06:55:34.031653 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:34.031660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:34.031730 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:34.031833 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:34.032567 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:34.033393 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:34.034658 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:34.035106 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:34.035241 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:34.038863 (+ 3622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41711: BlockingRecv error: recv got EOF from 127.0.97.3:41711 (error 108)
Metrics: {"server-negotiator.queue_time_us":167}
Sep 27 06:55:34 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:34 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:34.143944 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:34.135607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.1:47241)
0927 06:55:34.136090 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:34.136105 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:34.136254 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:34.136573 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:34.136578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:34.136993 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:34.137178 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:34.137186 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:34.138237 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:34.138242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:34.139151 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:34.139162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:34.139881 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:34.139887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:34.140063 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:34.140783 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:34.140800 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:34.143699 (+ 2899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":257,"threads_started":1}
W20260927 06:55:34.144119 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:34.135689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47241 (local address 127.0.97.62:42163)
0927 06:55:34.135932 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:34.135937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:34.136190 (+ 253us) server_negotiation.cc:407] Connection header received
0927 06:55:34.136373 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:34.136379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:34.136454 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:34.136550 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:34.137327 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:34.138111 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:34.139292 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:34.139742 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:34.139876 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:34.143879 (+ 4003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47241: BlockingRecv error: recv got EOF from 127.0.97.1:47241 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 06:55:34 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 06:55:34 dist-test-slave-f892 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 06:55:34.610668 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:34.598718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42163 (local address 127.0.97.2:44009)
0927 06:55:34.599210 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:34.599225 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:34.599333 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:34.599604 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:34.599607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:34.600090 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:34.600322 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:34.600335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:34.601681 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:34.601685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:34.602650 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:34.602661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:34.603356 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:34.603360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:34.603465 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:34.607485 (+ 4020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:34.607502 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:34.610412 (+ 2910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42163: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":275,"threads_started":1}
W20260927 06:55:34.610831 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:34.598759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44009 (local address 127.0.97.62:42163)
0927 06:55:34.598915 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:34.598919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:34.599294 (+ 375us) server_negotiation.cc:407] Connection header received
0927 06:55:34.599425 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:34.599429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:34.599481 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:34.599565 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:34.600468 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:34.601545 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:34.602784 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:34.603242 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:34.603333 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:34.610612 (+ 7279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44009: BlockingRecv error: recv got EOF from 127.0.97.2:44009 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 06:55:34.892930 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 9219
I20260927 06:55:34.904002 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:34.893933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42122 (local address 127.0.97.1:45723)
0927 06:55:34.894527 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:34.894535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:34.894561 (+ 26us) server_negotiation.cc:407] Connection header received
0927 06:55:34.894618 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:34.894625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:34.894853 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:34.895003 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:34.896280 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:34.897265 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:34.898452 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:34.898968 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:34.899072 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:34.901301 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:34.901309 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:34.901321 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:34.901348 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:34.903576 (+ 2228us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 06:55:34.903793 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42122: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":318,"threads_started":1}
W20260927 06:55:34.904121 9679 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42122: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 06:55:34.904213 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 9219: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:45723: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 06:55:34.904458 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 9219
I20260927 06:55:34.920953 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 9370
I20260927 06:55:34.932611 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:34.921950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37426 (local address 127.0.97.2:35307)
0927 06:55:34.922629 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:34.922636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:34.922673 (+ 37us) server_negotiation.cc:407] Connection header received
0927 06:55:34.922729 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:34.922736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:34.922949 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:34.923100 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:34.924261 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:34.925354 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:34.926656 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:34.927190 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:34.927304 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:34.929759 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:34.929768 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:34.929781 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:34.929818 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:34.932132 (+ 2314us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 06:55:34.932366 (+ 234us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37426: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":389,"threads_started":1}
W20260927 06:55:34.932741 9680 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37426: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 06:55:34.932816 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 9370: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:35307: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 06:55:34.933028 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 9370
I20260927 06:55:34.948563 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 9501
I20260927 06:55:34.959328 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:34.949354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.0.97.3:34701)
0927 06:55:34.949888 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:34.949895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:34.949920 (+ 25us) server_negotiation.cc:407] Connection header received
0927 06:55:34.949976 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:34.949982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:34.950245 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:34.950390 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:34.951499 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:34.952480 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:34.953579 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:34.954226 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:34.954324 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:34.956630 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:34.956638 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:34.956650 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:34.956691 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:34.958893 (+ 2202us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 06:55:34.959111 (+ 218us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47154: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":271,"threads_started":1}
W20260927 06:55:34.959445 9681 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47154: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 06:55:34.959455 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 9501: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:34701: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 06:55:34.959595 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 9501
I20260927 06:55:34.975731 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 9150
I20260927 06:55:34.977041 9212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:35.087816 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 9150
2026-09-27T06:55:35Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20910 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T06:55:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:55:35Z Disabled control of system clock
I20260927 06:55:35.160969 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42813
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 06:55:35.473305 9686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:35.473560 9686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:35.473634 9686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:35.480615 9686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:55:35.480702 9686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:35.480737 9686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:35.480789 9686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:55:35.480831 9686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:55:35.489449 9686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:35.490603 9686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:35.492326 9686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:35.503396 9694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:35.503502 9691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:35.503477 9692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:35.504273 9686 server_base.cc:1034] running on GCE node
I20260927 06:55:35.504964 9686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:35.506211 9686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:35.507400 9686 hybrid_clock.cc:648] HybridClock initialized: now 1790492135507346 us; error 54 us; skew 500 ppm
I20260927 06:55:35.507735 9686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:55:35.509752 9686 webserver.cc:460] Webserver started at http://127.0.97.62:35991/ using document root <none> and password file <none>
I20260927 06:55:35.510437 9686 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:35.510542 9686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:35.510859 9686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:35.513489 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "31e7b204f3ec40fabaf52dbe90fa6c99"
format_stamp: "Formatted at 2026-09-27 06:55:35 on dist-test-slave-f892"
server_key: "efe23e5d5e24f20401cd0089054fb048"
server_key_iv: "b4b4c017a1dd2b2efb682547d934549d"
server_key_version: "encryptionkey@0"
I20260927 06:55:35.514288 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "31e7b204f3ec40fabaf52dbe90fa6c99"
format_stamp: "Formatted at 2026-09-27 06:55:35 on dist-test-slave-f892"
server_key: "efe23e5d5e24f20401cd0089054fb048"
server_key_iv: "b4b4c017a1dd2b2efb682547d934549d"
server_key_version: "encryptionkey@0"
I20260927 06:55:35.518852 9686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 06:55:35.521859 9700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:35.523123 9686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 06:55:35.523293 9686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "31e7b204f3ec40fabaf52dbe90fa6c99"
format_stamp: "Formatted at 2026-09-27 06:55:35 on dist-test-slave-f892"
server_key: "efe23e5d5e24f20401cd0089054fb048"
server_key_iv: "b4b4c017a1dd2b2efb682547d934549d"
server_key_version: "encryptionkey@0"
I20260927 06:55:35.523486 9686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:35.549896 9686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:35.550701 9686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:35.550977 9686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:35.568105 9686 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42813
I20260927 06:55:35.568145 9751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42813 every 8 connection(s)
I20260927 06:55:35.569310 9686 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:55:35.572934 9752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:35.579715 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 9686
I20260927 06:55:35.580083 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:55:35.580852 388 external_mini_cluster.cc:1428] Setting key c5c81477740ed82e2be72aa32f659a62
I20260927 06:55:35.583720 9752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99: Bootstrap starting.
I20260927 06:55:35.587008 9752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:35.588025 9752 log.cc:824] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99: Log is configured to *not* fsync() on all Append() calls
I20260927 06:55:35.590857 9752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99: No bootstrap required, opened a new log
I20260927 06:55:35.594331 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:35.582909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.0.97.62:42813)
0927 06:55:35.583922 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:35.583933 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:35.583971 (+ 38us) server_negotiation.cc:407] Connection header received
0927 06:55:35.585597 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:35.585617 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:35.586060 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:35.586374 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:35.586959 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:35.588313 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:35.590073 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:35.590686 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:35.590805 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:35.591991 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:35.591998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:35.592009 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:55:35.592531 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:35.592626 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:55:35.592645 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:35.592734 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:35.593001 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 06:55:35.593352 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":353,"threads_started":1}
I20260927 06:55:35.598618 9752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e7b204f3ec40fabaf52dbe90fa6c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42813 } }
I20260927 06:55:35.598922 9752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:35.599032 9752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31e7b204f3ec40fabaf52dbe90fa6c99, State: Initialized, Role: FOLLOWER
I20260927 06:55:35.599520 9752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e7b204f3ec40fabaf52dbe90fa6c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42813 } }
I20260927 06:55:35.599767 9752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:55:35.599915 9752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:55:35.600081 9752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:35.602126 9752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e7b204f3ec40fabaf52dbe90fa6c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42813 } }
I20260927 06:55:35.602578 9752 leader_election.cc:302] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31e7b204f3ec40fabaf52dbe90fa6c99; no voters:
I20260927 06:55:35.603118 9752 leader_election.cc:288] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:55:35.603271 9757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:55:35.603917 9757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99 [term 1 LEADER]: Becoming Leader. State: Replica: 31e7b204f3ec40fabaf52dbe90fa6c99, State: Running, Role: LEADER
I20260927 06:55:35.604427 9757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e7b204f3ec40fabaf52dbe90fa6c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42813 } }
I20260927 06:55:35.605008 9752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:55:35.607101 9759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31e7b204f3ec40fabaf52dbe90fa6c99. Latest consensus state: current_term: 1 leader_uuid: "31e7b204f3ec40fabaf52dbe90fa6c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e7b204f3ec40fabaf52dbe90fa6c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42813 } } }
I20260927 06:55:35.607492 9759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:35.608568 9758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31e7b204f3ec40fabaf52dbe90fa6c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e7b204f3ec40fabaf52dbe90fa6c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42813 } } }
I20260927 06:55:35.608870 9758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:35.609892 9766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:55:35.614735 9766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:55:35.622995 9766 catalog_manager.cc:1347] Generated new cluster ID: 1097ff4b91a1401fb8510e90c004c64b
I20260927 06:55:35.623101 9766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:55:35.645366 9766 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:55:35.647034 9766 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:55:35.659029 9766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31e7b204f3ec40fabaf52dbe90fa6c99: Generated new TSK 0
I20260927 06:55:35.660033 9766 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:55:35.666421 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42813
--builtin_ntp_servers=127.0.97.20:39205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 06:55:35.979389 9776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:35.979677 9776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:35.979815 9776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:35.986836 9776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:35.986917 9776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:35.987106 9776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:55:35.995965 9776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:35.997064 9776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:35.998807 9776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:36.011232 9781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:36.011114 9784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:36.011183 9782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:36.012421 9776 server_base.cc:1034] running on GCE node
I20260927 06:55:36.013082 9776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:36.014170 9776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:36.015381 9776 hybrid_clock.cc:648] HybridClock initialized: now 1790492136015341 us; error 38 us; skew 500 ppm
I20260927 06:55:36.015803 9776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:55:36.018087 9776 webserver.cc:460] Webserver started at http://127.0.97.1:41923/ using document root <none> and password file <none>
I20260927 06:55:36.018791 9776 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:36.018899 9776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:36.019210 9776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:36.022190 9776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "086ec202368146cf9df87525c0eaaca5"
format_stamp: "Formatted at 2026-09-27 06:55:36 on dist-test-slave-f892"
server_key: "283e408cc335de10b02b80acf534ba0c"
server_key_iv: "987531eb6fc952a042d8f0d808ecb3e8"
server_key_version: "encryptionkey@0"
I20260927 06:55:36.022939 9776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "086ec202368146cf9df87525c0eaaca5"
format_stamp: "Formatted at 2026-09-27 06:55:36 on dist-test-slave-f892"
server_key: "283e408cc335de10b02b80acf534ba0c"
server_key_iv: "987531eb6fc952a042d8f0d808ecb3e8"
server_key_version: "encryptionkey@0"
I20260927 06:55:36.027843 9776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 06:55:36.031147 9790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:36.032459 9776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 06:55:36.032640 9776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "086ec202368146cf9df87525c0eaaca5"
format_stamp: "Formatted at 2026-09-27 06:55:36 on dist-test-slave-f892"
server_key: "283e408cc335de10b02b80acf534ba0c"
server_key_iv: "987531eb6fc952a042d8f0d808ecb3e8"
server_key_version: "encryptionkey@0"
I20260927 06:55:36.032851 9776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:36.054692 9776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:36.055514 9776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:36.055837 9776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:36.056718 9776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:36.058147 9776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:36.058274 9776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:36.058410 9776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:36.058485 9776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:36.089864 9776 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44609
I20260927 06:55:36.089957 9902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44609 every 8 connection(s)
I20260927 06:55:36.091167 9776 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:55:36.099303 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 9776
I20260927 06:55:36.099488 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:55:36.100124 388 external_mini_cluster.cc:1428] Setting key 02146aa6e91ff43a9a01aa86df1e9026
I20260927 06:55:36.102769 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42813
--builtin_ntp_servers=127.0.97.20:39205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 06:55:36.103211 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:36.093802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54375 (local address 127.0.97.62:42813)
0927 06:55:36.093984 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:36.093988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:36.095332 (+ 1344us) server_negotiation.cc:407] Connection header received
0927 06:55:36.096964 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:36.096968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:36.097031 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:36.097117 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:36.098141 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:36.098817 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:36.100507 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:36.100953 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:36.101092 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:36.102433 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:36.102443 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:36.102446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:55:36.102461 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:36.102488 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:55:36.102499 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:36.102590 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:36.102812 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 06:55:36.102897 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 06:55:36.104138 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:36.094307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42813 (local address 127.0.97.1:54375)
0927 06:55:36.095181 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:36.095226 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:36.096737 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:36.097318 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:36.097326 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:36.097404 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:36.097933 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:36.097948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:36.098974 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:36.098978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:36.100369 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:36.100381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:36.101174 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:36.101182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:36.101416 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:36.102133 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:55:36.102136 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:36.102173 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:55:36.102252 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:36.102590 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:36.102603 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:36.102613 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 06:55:36.102790 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 06:55:36.103153 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":269,"threads_started":1}
I20260927 06:55:36.105569 9903 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42813
I20260927 06:55:36.105873 9903 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:36.106614 9903 heartbeater.cc:511] Master 127.0.97.62:42813 requested a full tablet report, sending...
I20260927 06:55:36.108762 9717 ts_manager.cc:194] Registered new tserver with Master: 086ec202368146cf9df87525c0eaaca5 (127.0.97.1:44609)
I20260927 06:55:36.110427 9717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:54375
W20260927 06:55:36.415809 9906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:36.416097 9906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:36.416191 9906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:36.423389 9906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:36.423488 9906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:36.423588 9906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:55:36.433195 9906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:36.434455 9906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:36.436676 9906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:36.449291 9911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:36.449250 9914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:36.449348 9912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:36.454965 9906 server_base.cc:1034] running on GCE node
I20260927 06:55:36.455605 9906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:36.456560 9906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:36.457791 9906 hybrid_clock.cc:648] HybridClock initialized: now 1790492136457732 us; error 69 us; skew 500 ppm
I20260927 06:55:36.458161 9906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:55:36.460292 9906 webserver.cc:460] Webserver started at http://127.0.97.2:46463/ using document root <none> and password file <none>
I20260927 06:55:36.460968 9906 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:36.461093 9906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:36.461409 9906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:36.464233 9906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "72864e50b0bf4f4fa87e657a03f2a0ba"
format_stamp: "Formatted at 2026-09-27 06:55:36 on dist-test-slave-f892"
server_key: "09e8292cc74757815608fd696680afe5"
server_key_iv: "987ea795bf51bf8452c5e5e09d53ac31"
server_key_version: "encryptionkey@0"
I20260927 06:55:36.464996 9906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "72864e50b0bf4f4fa87e657a03f2a0ba"
format_stamp: "Formatted at 2026-09-27 06:55:36 on dist-test-slave-f892"
server_key: "09e8292cc74757815608fd696680afe5"
server_key_iv: "987ea795bf51bf8452c5e5e09d53ac31"
server_key_version: "encryptionkey@0"
I20260927 06:55:36.469704 9906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260927 06:55:36.472810 9920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:36.474123 9906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260927 06:55:36.474299 9906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "72864e50b0bf4f4fa87e657a03f2a0ba"
format_stamp: "Formatted at 2026-09-27 06:55:36 on dist-test-slave-f892"
server_key: "09e8292cc74757815608fd696680afe5"
server_key_iv: "987ea795bf51bf8452c5e5e09d53ac31"
server_key_version: "encryptionkey@0"
I20260927 06:55:36.474495 9906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:36.487615 9906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:36.488353 9906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:36.488632 9906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:36.489496 9906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:36.490890 9906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:36.491008 9906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:36.491139 9906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:36.491225 9906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:36.523679 9906 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45767
I20260927 06:55:36.523769 10032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45767 every 8 connection(s)
I20260927 06:55:36.524914 9906 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:55:36.535118 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 9906
I20260927 06:55:36.535308 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:55:36.535841 388 external_mini_cluster.cc:1428] Setting key 23c20306ed6d7dab7c22d7434caa85cf
I20260927 06:55:36.537155 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:36.527316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58891 (local address 127.0.97.62:42813)
0927 06:55:36.527506 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:36.527510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:36.528891 (+ 1381us) server_negotiation.cc:407] Connection header received
0927 06:55:36.530619 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:36.530624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:36.530690 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:36.530779 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:36.531781 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:36.532483 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:36.533985 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:36.534438 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:36.534537 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:36.536214 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:36.536219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:36.536222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:55:36.536239 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:36.536267 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:55:36.536277 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:36.536345 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:36.536621 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 06:55:36.536751 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 06:55:36.538167 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:36.527858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42813 (local address 127.0.97.2:58891)
0927 06:55:36.528737 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:36.528776 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:36.530310 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:36.530971 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:36.530978 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:36.531057 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:36.531584 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:36.531602 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:36.532606 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:36.532610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:36.533859 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:36.533871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:36.534716 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:36.534728 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:36.535052 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:36.535883 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:55:36.535889 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:36.535940 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:55:36.536037 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:36.536379 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:36.536389 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:36.536397 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 06:55:36.536579 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 06:55:36.536905 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":289,"threads_started":1}
I20260927 06:55:36.538839 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42813
--builtin_ntp_servers=127.0.97.20:39205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 06:55:36.539510 10033 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42813
I20260927 06:55:36.539820 10033 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:36.540732 10033 heartbeater.cc:511] Master 127.0.97.62:42813 requested a full tablet report, sending...
I20260927 06:55:36.542241 9717 ts_manager.cc:194] Registered new tserver with Master: 72864e50b0bf4f4fa87e657a03f2a0ba (127.0.97.2:45767)
I20260927 06:55:36.543028 9717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:58891
W20260927 06:55:36.852404 10036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:36.852713 10036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:36.852799 10036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:36.860108 10036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:36.860224 10036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:36.860334 10036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:55:36.869168 10036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:36.870445 10036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:36.872146 10036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:36.887708 10041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:36.887674 10044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:36.888096 10042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:36.888471 10036 server_base.cc:1034] running on GCE node
I20260927 06:55:36.889045 10036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:36.889969 10036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:36.891250 10036 hybrid_clock.cc:648] HybridClock initialized: now 1790492136891188 us; error 56 us; skew 500 ppm
I20260927 06:55:36.891564 10036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:55:36.893963 10036 webserver.cc:460] Webserver started at http://127.0.97.3:40671/ using document root <none> and password file <none>
I20260927 06:55:36.894711 10036 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:36.894850 10036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:36.895177 10036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:36.897913 10036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "6796f1936bd94b2abb97427f120da7c8"
format_stamp: "Formatted at 2026-09-27 06:55:36 on dist-test-slave-f892"
server_key: "60973ddd8f2b7c0d875657620e748956"
server_key_iv: "6f3d25a240555939a71f61a8d0c135f6"
server_key_version: "encryptionkey@0"
I20260927 06:55:36.898751 10036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6796f1936bd94b2abb97427f120da7c8"
format_stamp: "Formatted at 2026-09-27 06:55:36 on dist-test-slave-f892"
server_key: "60973ddd8f2b7c0d875657620e748956"
server_key_iv: "6f3d25a240555939a71f61a8d0c135f6"
server_key_version: "encryptionkey@0"
I20260927 06:55:36.903541 10036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 06:55:36.906826 10050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:36.908058 10036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 06:55:36.908241 10036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "6796f1936bd94b2abb97427f120da7c8"
format_stamp: "Formatted at 2026-09-27 06:55:36 on dist-test-slave-f892"
server_key: "60973ddd8f2b7c0d875657620e748956"
server_key_iv: "6f3d25a240555939a71f61a8d0c135f6"
server_key_version: "encryptionkey@0"
I20260927 06:55:36.908450 10036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:36.923048 10036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:36.923784 10036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:36.924044 10036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:36.924885 10036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:36.926306 10036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:36.926419 10036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:36.926553 10036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:36.926640 10036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:36.956308 10036 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44087
I20260927 06:55:36.956347 10162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44087 every 8 connection(s)
I20260927 06:55:36.957475 10036 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:55:36.960410 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 10036
I20260927 06:55:36.960569 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:55:36.961004 388 external_mini_cluster.cc:1428] Setting key 4abd17f7a5015627ad7c7d48245ea37c
I20260927 06:55:36.969094 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:36.959855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51715 (local address 127.0.97.62:42813)
0927 06:55:36.960023 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:36.960027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:36.961455 (+ 1428us) server_negotiation.cc:407] Connection header received
0927 06:55:36.963472 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:36.963477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:36.963558 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:36.963653 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:36.964590 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:36.965250 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:36.966873 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:36.967243 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:36.967333 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:36.968432 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:36.968436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:36.968440 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:55:36.968455 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:36.968483 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:55:36.968494 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:36.968562 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:36.968818 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 06:55:36.968884 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 06:55:36.970134 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:36.960299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42813 (local address 127.0.97.3:51715)
0927 06:55:36.961309 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:36.961350 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:36.963187 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:36.963882 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:36.963889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:36.963966 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:36.964463 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:36.964477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:36.965365 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:36.965369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:36.966757 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:36.966768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:36.967404 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:36.967408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:36.967580 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:36.968165 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:55:36.968167 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:36.968220 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:55:36.968283 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:36.968622 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:36.968630 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:36.968642 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 06:55:36.968797 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 06:55:36.969048 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":371,"threads_started":1}
I20260927 06:55:36.971357 10163 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42813
I20260927 06:55:36.971658 10163 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:36.972404 10163 heartbeater.cc:511] Master 127.0.97.62:42813 requested a full tablet report, sending...
I20260927 06:55:36.973758 9717 ts_manager.cc:194] Registered new tserver with Master: 6796f1936bd94b2abb97427f120da7c8 (127.0.97.3:44087)
I20260927 06:55:36.974601 9717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:51715
I20260927 06:55:36.975996 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 06:55:36.982327 9755 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:36.980663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57384 (local address 127.0.97.62:42813)
0927 06:55:36.980853 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:36.980861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:36.981226 (+ 365us) server_negotiation.cc:407] Connection header received
0927 06:55:36.981415 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:36.981420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:36.981502 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:36.981610 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:36.982046 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57384: BlockingRecv error: recv got EOF from 127.0.0.1:57384 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 06:55:36.983140 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 9776
I20260927 06:55:36.989929 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:36.983989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39634 (local address 127.0.97.1:44609)
0927 06:55:36.984521 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:36.984528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:36.984552 (+ 24us) server_negotiation.cc:407] Connection header received
0927 06:55:36.984602 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:36.984626 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:36.984801 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:36.984917 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:36.985243 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:36.986239 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:36.987325 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:36.987802 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:36.987892 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:36.988757 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:36.988764 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:36.988774 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:55:36.989313 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:36.989356 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:55:36.989373 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:36.989455 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:36.989567 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 06:55:36.989661 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":275,"threads_started":1}
I20260927 06:55:36.991040 9898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:37.070580 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 9776
I20260927 06:55:37.087041 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 9906
I20260927 06:55:37.094931 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:37.087910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50062 (local address 127.0.97.2:45767)
0927 06:55:37.088507 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:37.088514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:37.088539 (+ 25us) server_negotiation.cc:407] Connection header received
0927 06:55:37.088603 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:37.088609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:37.088877 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:37.089045 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:37.089539 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:37.090688 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:37.091870 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:37.092452 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:37.092553 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:37.093528 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:37.093535 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:37.093547 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:55:37.094237 (+ 690us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:37.094282 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:55:37.094307 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:37.094395 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:37.094530 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 06:55:37.094649 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":333,"threads_started":1}
I20260927 06:55:37.096125 10028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:37.177258 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 9906
I20260927 06:55:37.192869 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 10036
I20260927 06:55:37.200879 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:37.193867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.0.97.3:44087)
0927 06:55:37.194471 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:37.194483 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:37.194524 (+ 41us) server_negotiation.cc:407] Connection header received
0927 06:55:37.194589 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:37.194595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:37.194797 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:37.194963 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:37.195397 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:37.196418 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:37.197639 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:37.198256 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:37.198399 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:37.199531 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:37.199538 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:37.199550 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:55:37.200073 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:37.200119 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:55:37.200136 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:37.200253 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:37.200393 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 06:55:37.200524 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":339,"threads_started":1}
I20260927 06:55:37.202198 10158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:37.282212 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 10036
I20260927 06:55:37.298161 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 9686
I20260927 06:55:37.299340 9747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:37.398051 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 9686
2026-09-27T06:55:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2304 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T06:55:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:55:37Z Disabled control of system clock
I20260927 06:55:37.467504 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36753
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 06:55:37.777189 10182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:37.777491 10182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:37.777576 10182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:37.784914 10182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:55:37.785017 10182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:37.785053 10182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:37.785101 10182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:55:37.785144 10182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:55:37.794276 10182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36753
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:37.795670 10182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:37.797328 10182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:37.808570 10187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:37.808526 10188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:37.808604 10190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:37.809348 10182 server_base.cc:1034] running on GCE node
I20260927 06:55:37.810076 10182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:37.811331 10182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:37.812533 10182 hybrid_clock.cc:648] HybridClock initialized: now 1790492137812498 us; error 33 us; skew 500 ppm
I20260927 06:55:37.812907 10182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:55:37.815207 10182 webserver.cc:460] Webserver started at http://127.0.97.62:34135/ using document root <none> and password file <none>
I20260927 06:55:37.815874 10182 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:37.815992 10182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:37.816314 10182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:37.819203 10182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "b7e14f14257d420993e1275dda3e3f32"
format_stamp: "Formatted at 2026-09-27 06:55:37 on dist-test-slave-f892"
server_key: "c6cec78fcb31daee1202b0888fafd683"
server_key_iv: "167fbb7a0da78d9d3e0773e476c62223"
server_key_version: "encryptionkey@0"
I20260927 06:55:37.820021 10182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "b7e14f14257d420993e1275dda3e3f32"
format_stamp: "Formatted at 2026-09-27 06:55:37 on dist-test-slave-f892"
server_key: "c6cec78fcb31daee1202b0888fafd683"
server_key_iv: "167fbb7a0da78d9d3e0773e476c62223"
server_key_version: "encryptionkey@0"
I20260927 06:55:37.824923 10182 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 06:55:37.828051 10196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:37.829362 10182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 06:55:37.829557 10182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "b7e14f14257d420993e1275dda3e3f32"
format_stamp: "Formatted at 2026-09-27 06:55:37 on dist-test-slave-f892"
server_key: "c6cec78fcb31daee1202b0888fafd683"
server_key_iv: "167fbb7a0da78d9d3e0773e476c62223"
server_key_version: "encryptionkey@0"
I20260927 06:55:37.829808 10182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:37.835089 10182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:37.835794 10182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:37.836079 10182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:37.853371 10182 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36753
I20260927 06:55:37.853405 10247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36753 every 8 connection(s)
I20260927 06:55:37.854763 10182 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:55:37.858227 10248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:37.864912 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 10182
I20260927 06:55:37.865128 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:55:37.865756 388 external_mini_cluster.cc:1428] Setting key ece4eda5e11bf0c438289aa2a585fca9
I20260927 06:55:37.868152 10248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32: Bootstrap starting.
I20260927 06:55:37.871160 10248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:37.872459 10248 log.cc:824] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32: Log is configured to *not* fsync() on all Append() calls
I20260927 06:55:37.873838 10251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:37.867730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.0.97.62:36753)
0927 06:55:37.868769 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:37.868779 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:37.868817 (+ 38us) server_negotiation.cc:407] Connection header received
0927 06:55:37.870419 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:37.870438 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:37.870788 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:37.871120 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:37.872256 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:37.872262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:37.872275 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:55:37.872310 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:37.872363 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:55:37.872379 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:37.872535 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:37.872752 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 06:55:37.873002 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":424,"threads_started":1}
I20260927 06:55:37.875299 10248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32: No bootstrap required, opened a new log
I20260927 06:55:37.882733 10248 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e14f14257d420993e1275dda3e3f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36753 } }
I20260927 06:55:37.883018 10248 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:37.883088 10248 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7e14f14257d420993e1275dda3e3f32, State: Initialized, Role: FOLLOWER
I20260927 06:55:37.883546 10248 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e14f14257d420993e1275dda3e3f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36753 } }
I20260927 06:55:37.883739 10248 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:55:37.883827 10248 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:55:37.883939 10248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:37.885967 10248 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e14f14257d420993e1275dda3e3f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36753 } }
I20260927 06:55:37.886535 10248 leader_election.cc:302] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7e14f14257d420993e1275dda3e3f32; no voters:
I20260927 06:55:37.887117 10248 leader_election.cc:288] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:55:37.887272 10253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:55:37.888019 10253 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32 [term 1 LEADER]: Becoming Leader. State: Replica: b7e14f14257d420993e1275dda3e3f32, State: Running, Role: LEADER
I20260927 06:55:37.888520 10253 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e14f14257d420993e1275dda3e3f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36753 } }
I20260927 06:55:37.889027 10248 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:55:37.891376 10254 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7e14f14257d420993e1275dda3e3f32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e14f14257d420993e1275dda3e3f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36753 } } }
I20260927 06:55:37.891983 10254 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:37.891356 10255 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7e14f14257d420993e1275dda3e3f32. Latest consensus state: current_term: 1 leader_uuid: "b7e14f14257d420993e1275dda3e3f32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e14f14257d420993e1275dda3e3f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36753 } } }
I20260927 06:55:37.892349 10255 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:37.899703 10262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:55:37.904047 10262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:55:37.911669 10262 catalog_manager.cc:1347] Generated new cluster ID: 2f5d31cc31104f30bedc62bfb50e6d4a
I20260927 06:55:37.911774 10262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:55:37.930059 10262 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:55:37.930526 10262 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:55:37.947197 10262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7e14f14257d420993e1275dda3e3f32: Generated new TSK 0
I20260927 06:55:37.948160 10262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:55:37.952369 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36753
--builtin_ntp_servers=127.0.97.20:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 06:55:38.269510 10272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:38.269763 10272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:38.269920 10272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:38.276966 10272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:38.277052 10272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:38.277226 10272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:55:38.285948 10272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:38.287164 10272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:38.288866 10272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:38.301746 10277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:38.301819 10278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:38.301795 10280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:38.302839 10272 server_base.cc:1034] running on GCE node
I20260927 06:55:38.303395 10272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:38.304229 10272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:38.305413 10272 hybrid_clock.cc:648] HybridClock initialized: now 1790492138305366 us; error 45 us; skew 500 ppm
I20260927 06:55:38.305722 10272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:55:38.307771 10272 webserver.cc:460] Webserver started at http://127.0.97.1:40271/ using document root <none> and password file <none>
I20260927 06:55:38.308403 10272 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:38.308514 10272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:38.308830 10272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:38.311497 10272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "61082383c6d546ca882f974b16a22309"
format_stamp: "Formatted at 2026-09-27 06:55:38 on dist-test-slave-f892"
server_key: "4e6ad191eafcc46a9f08c51055831b06"
server_key_iv: "6b58804917cb27f27c4b848cb37fd576"
server_key_version: "encryptionkey@0"
I20260927 06:55:38.312256 10272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "61082383c6d546ca882f974b16a22309"
format_stamp: "Formatted at 2026-09-27 06:55:38 on dist-test-slave-f892"
server_key: "4e6ad191eafcc46a9f08c51055831b06"
server_key_iv: "6b58804917cb27f27c4b848cb37fd576"
server_key_version: "encryptionkey@0"
I20260927 06:55:38.316790 10272 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 06:55:38.319898 10286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:38.321061 10272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 06:55:38.321225 10272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "61082383c6d546ca882f974b16a22309"
format_stamp: "Formatted at 2026-09-27 06:55:38 on dist-test-slave-f892"
server_key: "4e6ad191eafcc46a9f08c51055831b06"
server_key_iv: "6b58804917cb27f27c4b848cb37fd576"
server_key_version: "encryptionkey@0"
I20260927 06:55:38.321434 10272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:38.326592 10272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:38.327312 10272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:38.327585 10272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:38.328441 10272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:38.329712 10272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:38.329826 10272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:38.329957 10272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:38.330130 10272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:38.361394 10272 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35929
I20260927 06:55:38.361447 10398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35929 every 8 connection(s)
I20260927 06:55:38.362852 10272 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:55:38.370680 10251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:38.365342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34879 (local address 127.0.97.62:36753)
0927 06:55:38.365502 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:38.365507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:38.366796 (+ 1289us) server_negotiation.cc:407] Connection header received
0927 06:55:38.368456 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:38.368461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:38.368533 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:38.368649 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:38.369928 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:38.369934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:38.369939 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:55:38.369965 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:38.369996 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:55:38.370063 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:38.370179 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:38.370356 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 06:55:38.370423 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 06:55:38.371572 10401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:38.365731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36753 (local address 127.0.97.1:34879)
0927 06:55:38.366617 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:38.366680 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:38.368192 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:38.368863 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:38.368871 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:38.368953 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:38.369641 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:55:38.369643 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:38.369676 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:55:38.369754 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:38.370141 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:38.370150 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:38.370159 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 06:55:38.370434 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 06:55:38.370657 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":294,"threads_started":1}
I20260927 06:55:38.372399 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 10272
I20260927 06:55:38.372627 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:55:38.373117 10399 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36753
I20260927 06:55:38.373430 10399 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:38.373523 388 external_mini_cluster.cc:1428] Setting key 6440fbbbc0d6ee40b522ef3a7fa9312c
I20260927 06:55:38.374191 10399 heartbeater.cc:511] Master 127.0.97.62:36753 requested a full tablet report, sending...
I20260927 06:55:38.376494 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36753
--builtin_ntp_servers=127.0.97.20:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 06:55:38.376533 10213 ts_manager.cc:194] Registered new tserver with Master: 61082383c6d546ca882f974b16a22309 (127.0.97.1:35929)
W20260927 06:55:38.688882 10402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:38.689211 10402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:38.689355 10402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:38.696615 10402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:38.696717 10402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:38.696918 10402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:55:38.706316 10402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:38.707541 10402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:38.709249 10402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:38.721799 10407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:38.721832 10408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:38.721726 10410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:38.722752 10402 server_base.cc:1034] running on GCE node
I20260927 06:55:38.723376 10402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:38.724349 10402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:38.725600 10402 hybrid_clock.cc:648] HybridClock initialized: now 1790492138725547 us; error 50 us; skew 500 ppm
I20260927 06:55:38.725971 10402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:55:38.728233 10402 webserver.cc:460] Webserver started at http://127.0.97.2:40397/ using document root <none> and password file <none>
I20260927 06:55:38.728890 10402 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:38.729015 10402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:38.729317 10402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:38.733676 10402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "41a93483ecba441d9f1fb73d61350870"
format_stamp: "Formatted at 2026-09-27 06:55:38 on dist-test-slave-f892"
server_key: "789b6ec9cbc35112882f8a65c448b6d2"
server_key_iv: "03a6b08cf41b7b8603226ccc794ace34"
server_key_version: "encryptionkey@0"
I20260927 06:55:38.734588 10402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "41a93483ecba441d9f1fb73d61350870"
format_stamp: "Formatted at 2026-09-27 06:55:38 on dist-test-slave-f892"
server_key: "789b6ec9cbc35112882f8a65c448b6d2"
server_key_iv: "03a6b08cf41b7b8603226ccc794ace34"
server_key_version: "encryptionkey@0"
I20260927 06:55:38.739444 10402 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 06:55:38.742624 10416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:38.743854 10402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:55:38.744022 10402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "41a93483ecba441d9f1fb73d61350870"
format_stamp: "Formatted at 2026-09-27 06:55:38 on dist-test-slave-f892"
server_key: "789b6ec9cbc35112882f8a65c448b6d2"
server_key_iv: "03a6b08cf41b7b8603226ccc794ace34"
server_key_version: "encryptionkey@0"
I20260927 06:55:38.744243 10402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:38.749468 10402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:38.750272 10402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:38.750512 10402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:38.751368 10402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:38.752646 10402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:38.752727 10402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:38.752867 10402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:38.752957 10402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:38.784703 10402 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36149
I20260927 06:55:38.784751 10528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36149 every 8 connection(s)
I20260927 06:55:38.785965 10402 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:55:38.786993 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 10402
I20260927 06:55:38.787189 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:55:38.787690 388 external_mini_cluster.cc:1428] Setting key 52b144e3e1e97b38a205a04fee629cf8
I20260927 06:55:38.790796 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36753
--builtin_ntp_servers=127.0.97.20:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 06:55:38.795648 10251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:38.788919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36207 (local address 127.0.97.62:36753)
0927 06:55:38.789117 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:38.789125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:38.790836 (+ 1711us) server_negotiation.cc:407] Connection header received
0927 06:55:38.793459 (+ 2623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:38.793463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:38.793511 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:38.793601 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:38.795002 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:38.795006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:38.795009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:55:38.795024 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:38.795054 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:55:38.795066 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:38.795169 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:38.795390 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 06:55:38.795438 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 06:55:38.796866 10531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:38.789583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36753 (local address 127.0.97.2:36207)
0927 06:55:38.790672 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:38.790723 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:38.793192 (+ 2469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:38.793854 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:38.793866 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:38.793979 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:38.794680 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:55:38.794686 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:38.794735 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:55:38.794840 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:38.795177 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:38.795189 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:38.795200 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 06:55:38.795457 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 06:55:38.795695 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":341,"threads_started":1}
I20260927 06:55:38.798192 10529 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36753
I20260927 06:55:38.798477 10529 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:38.799113 10529 heartbeater.cc:511] Master 127.0.97.62:36753 requested a full tablet report, sending...
I20260927 06:55:38.800390 10213 ts_manager.cc:194] Registered new tserver with Master: 41a93483ecba441d9f1fb73d61350870 (127.0.97.2:36149)
W20260927 06:55:39.117058 10532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:39.117354 10532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:39.117524 10532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:39.124784 10532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:39.124910 10532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:39.125085 10532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:55:39.134385 10532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:39.135682 10532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:39.137442 10532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:39.149847 10537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:39.149871 10540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:39.149902 10538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:39.150687 10532 server_base.cc:1034] running on GCE node
I20260927 06:55:39.151228 10532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:39.152094 10532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:39.153318 10532 hybrid_clock.cc:648] HybridClock initialized: now 1790492139153268 us; error 39 us; skew 500 ppm
I20260927 06:55:39.153607 10532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:55:39.155755 10532 webserver.cc:460] Webserver started at http://127.0.97.3:32779/ using document root <none> and password file <none>
I20260927 06:55:39.156386 10532 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:39.156514 10532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:39.156822 10532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:39.159535 10532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "62853571bfe74f1bafa57540b9b403db"
format_stamp: "Formatted at 2026-09-27 06:55:39 on dist-test-slave-f892"
server_key: "ba248cc89455d09db07a1f1bc53de074"
server_key_iv: "0c853a432685993d5ab0d7c14ddbf719"
server_key_version: "encryptionkey@0"
I20260927 06:55:39.160284 10532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "62853571bfe74f1bafa57540b9b403db"
format_stamp: "Formatted at 2026-09-27 06:55:39 on dist-test-slave-f892"
server_key: "ba248cc89455d09db07a1f1bc53de074"
server_key_iv: "0c853a432685993d5ab0d7c14ddbf719"
server_key_version: "encryptionkey@0"
I20260927 06:55:39.164806 10532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 06:55:39.167848 10546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:39.169073 10532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 06:55:39.169265 10532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "62853571bfe74f1bafa57540b9b403db"
format_stamp: "Formatted at 2026-09-27 06:55:39 on dist-test-slave-f892"
server_key: "ba248cc89455d09db07a1f1bc53de074"
server_key_iv: "0c853a432685993d5ab0d7c14ddbf719"
server_key_version: "encryptionkey@0"
I20260927 06:55:39.169483 10532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:39.175016 10532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:39.175817 10532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:39.176096 10532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:39.176970 10532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:39.178369 10532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:39.178493 10532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:39.178625 10532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:39.178712 10532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:39.209854 10532 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42129
I20260927 06:55:39.209903 10658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42129 every 8 connection(s)
I20260927 06:55:39.211139 10532 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:55:39.214707 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 10532
I20260927 06:55:39.214891 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:55:39.215371 388 external_mini_cluster.cc:1428] Setting key 900ea6e2be7ffab79a503531ef17ca5e
I20260927 06:55:39.219498 10251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:39.213528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57987 (local address 127.0.97.62:36753)
0927 06:55:39.213709 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:39.213713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:39.215170 (+ 1457us) server_negotiation.cc:407] Connection header received
0927 06:55:39.217248 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:39.217254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:39.217319 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:39.217432 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:39.218808 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:39.218813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:39.218818 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:55:39.218841 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:39.218870 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:55:39.218881 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:39.218993 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:39.219182 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 06:55:39.219248 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 06:55:39.220378 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:39.213969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36753 (local address 127.0.97.3:57987)
0927 06:55:39.215006 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:39.215060 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:39.216966 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:39.217664 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:39.217672 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:39.217759 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:39.218519 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:55:39.218522 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:39.218556 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:55:39.218649 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:39.218968 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:39.218978 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:39.218985 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 06:55:39.219261 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 06:55:39.219507 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":286,"threads_started":1}
I20260927 06:55:39.221506 10659 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36753
I20260927 06:55:39.221753 10659 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:39.222290 10659 heartbeater.cc:511] Master 127.0.97.62:36753 requested a full tablet report, sending...
I20260927 06:55:39.223495 10213 ts_manager.cc:194] Registered new tserver with Master: 62853571bfe74f1bafa57540b9b403db (127.0.97.3:42129)
I20260927 06:55:39.230125 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 06:55:39.236152 10251 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:55:39.234664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.0.97.62:36753)
0927 06:55:39.234806 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:39.234810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:39.235324 (+ 514us) server_negotiation.cc:407] Connection header received
0927 06:55:39.235446 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:39.235457 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:39.235531 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:39.235619 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:39.235902 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43960: BlockingRecv error: recv got EOF from 127.0.0.1:43960 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 06:55:39.236728 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 10272
I20260927 06:55:39.239748 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:39.237479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35640 (local address 127.0.97.1:35929)
0927 06:55:39.237944 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:39.237950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:39.237973 (+ 23us) server_negotiation.cc:407] Connection header received
0927 06:55:39.238070 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:39.238077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:39.238257 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:39.238371 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:39.239223 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:39.239230 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:39.239240 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:55:39.239253 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:39.239295 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:55:39.239308 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:39.239411 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:39.239476 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 06:55:39.239547 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":227,"threads_started":1}
I20260927 06:55:39.240772 10394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:39.319931 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 10272
I20260927 06:55:39.336156 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 10402
I20260927 06:55:39.339684 10670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:39.337047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56350 (local address 127.0.97.2:36149)
0927 06:55:39.337507 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:39.337513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:39.337538 (+ 25us) server_negotiation.cc:407] Connection header received
0927 06:55:39.337594 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:39.337600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:39.337828 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:39.338000 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:39.338985 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:39.338991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:39.339002 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:55:39.339017 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:39.339053 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:55:39.339065 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:39.339205 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:39.339308 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 06:55:39.339402 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":229,"threads_started":1}
I20260927 06:55:39.340893 10524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:39.424970 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 10402
I20260927 06:55:39.440703 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 10532
I20260927 06:55:39.444661 10672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:39.441699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57428 (local address 127.0.97.3:42129)
0927 06:55:39.442349 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:39.442356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:39.442385 (+ 29us) server_negotiation.cc:407] Connection header received
0927 06:55:39.442442 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:39.442449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:39.442672 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:39.442837 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:39.443932 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:39.443939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:39.443949 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:55:39.443959 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:39.443996 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:55:39.444008 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:39.444185 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:39.444270 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 06:55:39.444364 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":368,"threads_started":1}
I20260927 06:55:39.445984 10654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:39.527781 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 10532
I20260927 06:55:39.543951 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 10182
I20260927 06:55:39.545099 10243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:39.642745 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 10182
2026-09-27T06:55:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2245 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:55:39 dist-test-slave-f892 krb5kdc[10677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:55:39 dist-test-slave-f892 krb5kdc[10677](info): set up 2 sockets
Sep 27 06:55:39 dist-test-slave-f892 krb5kdc[10677](info): commencing operation
krb5kdc: starting...
W20260927 06:55:40.762737 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:55:40 dist-test-slave-f892 krb5kdc[10677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:55:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:55:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:55:40.981040 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34697
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:55:41.298892 10690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:41.299180 10690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:41.299341 10690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:41.306463 10690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:55:41.306582 10690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:41.306685 10690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:41.306782 10690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:55:41.306861 10690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:55:41.315989 10690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:41.317256 10690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:41.318526 10690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:41.329679 10695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:41.329674 10696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:41.330116 10698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:41.330410 10690 server_base.cc:1034] running on GCE node
I20260927 06:55:41.331110 10690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:41.332460 10690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:41.333709 10690 hybrid_clock.cc:648] HybridClock initialized: now 1790492141333669 us; error 32 us; skew 500 ppm
I20260927 06:55:41.334161 10690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:41 dist-test-slave-f892 krb5kdc[10677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:41.337123 10690 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:55:41.338532 10690 webserver.cc:460] Webserver started at http://127.0.97.62:35017/ using document root <none> and password file <none>
I20260927 06:55:41.339141 10690 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:41.339238 10690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:41.339475 10690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:41.342430 10690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "d97c6585f5864c09a9a61338d631f3a8"
format_stamp: "Formatted at 2026-09-27 06:55:41 on dist-test-slave-f892"
server_key: "2516fe221fc0531448b6b9a657f90104"
server_key_iv: "1b7530680949d2bfba9c942c7bd7fb54"
server_key_version: "encryptionkey@0"
I20260927 06:55:41.343127 10690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "d97c6585f5864c09a9a61338d631f3a8"
format_stamp: "Formatted at 2026-09-27 06:55:41 on dist-test-slave-f892"
server_key: "2516fe221fc0531448b6b9a657f90104"
server_key_iv: "1b7530680949d2bfba9c942c7bd7fb54"
server_key_version: "encryptionkey@0"
I20260927 06:55:41.347905 10690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 06:55:41.351157 10705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:41.352408 10690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 06:55:41.352589 10690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "d97c6585f5864c09a9a61338d631f3a8"
format_stamp: "Formatted at 2026-09-27 06:55:41 on dist-test-slave-f892"
server_key: "2516fe221fc0531448b6b9a657f90104"
server_key_iv: "1b7530680949d2bfba9c942c7bd7fb54"
server_key_version: "encryptionkey@0"
I20260927 06:55:41.352766 10690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:41.377516 10690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:41.381515 10690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:41.381799 10690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:41.399631 10690 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34697
I20260927 06:55:41.399662 10756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34697 every 8 connection(s)
I20260927 06:55:41.400892 10690 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:55:41.402884 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 10690
I20260927 06:55:41.403110 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:55:41.403874 388 external_mini_cluster.cc:1428] Setting key 0f3cd40835ea793e629c938c7dd32b2e
I20260927 06:55:41.405185 10757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 06:55:41 dist-test-slave-f892 krb5kdc[10677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:55:41.419267 10757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8: Bootstrap starting.
I20260927 06:55:41.422521 10757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:41.423540 10757 log.cc:824] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8: Log is configured to *not* fsync() on all Append() calls
I20260927 06:55:41.426401 10757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8: No bootstrap required, opened a new log
I20260927 06:55:41.426811 10760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:41.405870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43466 (local address 127.0.97.62:34697)
0927 06:55:41.406658 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:41.406669 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:41.406707 (+ 38us) server_negotiation.cc:407] Connection header received
0927 06:55:41.408138 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:41.408158 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:41.408511 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:41.408805 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:41.410544 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:41.411858 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:41.413628 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:41.414379 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:41.414512 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:41.419353 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:41.419361 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:41.419378 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:41.419453 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:41.422904 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:41.423589 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:41.423596 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:41.423602 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:41.423799 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:41.424175 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:41.424184 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:41.424187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:41.424676 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:41.424961 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:41.425383 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 06:55:41.425792 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":397,"threads_started":1}
I20260927 06:55:41.434362 10757 raft_consensus.cc:348] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97c6585f5864c09a9a61338d631f3a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } }
I20260927 06:55:41.434698 10757 raft_consensus.cc:374] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:41.434818 10757 raft_consensus.cc:729] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97c6585f5864c09a9a61338d631f3a8, State: Initialized, Role: FOLLOWER
I20260927 06:55:41.435333 10757 consensus_queue.cc:260] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97c6585f5864c09a9a61338d631f3a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } }
I20260927 06:55:41.435627 10757 raft_consensus.cc:388] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:55:41.435789 10757 raft_consensus.cc:482] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:55:41.435979 10757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:41.437973 10757 raft_consensus.cc:504] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97c6585f5864c09a9a61338d631f3a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } }
I20260927 06:55:41.438586 10757 leader_election.cc:302] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d97c6585f5864c09a9a61338d631f3a8; no voters:
I20260927 06:55:41.439186 10757 leader_election.cc:288] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:55:41.439292 10762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:55:41.440008 10762 raft_consensus.cc:686] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8 [term 1 LEADER]: Becoming Leader. State: Replica: d97c6585f5864c09a9a61338d631f3a8, State: Running, Role: LEADER
I20260927 06:55:41.440519 10762 consensus_queue.cc:237] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97c6585f5864c09a9a61338d631f3a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } }
I20260927 06:55:41.441155 10757 sys_catalog.cc:564] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:55:41.444182 10764 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d97c6585f5864c09a9a61338d631f3a8. Latest consensus state: current_term: 1 leader_uuid: "d97c6585f5864c09a9a61338d631f3a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97c6585f5864c09a9a61338d631f3a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } }
I20260927 06:55:41.444466 10764 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:41.449945 10763 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d97c6585f5864c09a9a61338d631f3a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97c6585f5864c09a9a61338d631f3a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } }
I20260927 06:55:41.450330 10763 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8 [sys.catalog]: This master's current role is: LEADER
I20260927 06:55:41.452692 10771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:55:41.458933 10771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:55:41.466733 10771 catalog_manager.cc:1347] Generated new cluster ID: 00eed388ce3e4b68995bbe5782040b9e
I20260927 06:55:41.466842 10771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:55:41.493995 10771 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:55:41.495918 10771 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:55:41.505963 10771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8: Generated new TSK 0
I20260927 06:55:41.506940 10771 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 06:55:41.571288 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34697
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:55:41.888307 10783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:41.888551 10783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:41.888641 10783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:41.895694 10783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:41.895812 10783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:41.895923 10783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:55:41.905490 10783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:41.906705 10783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:41.907898 10783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:41.920543 10788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:41.920512 10791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:41.920557 10789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:41.921429 10783 server_base.cc:1034] running on GCE node
I20260927 06:55:41.922125 10783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:41.923008 10783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:41.924265 10783 hybrid_clock.cc:648] HybridClock initialized: now 1790492141924219 us; error 38 us; skew 500 ppm
I20260927 06:55:41.924597 10783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:41 dist-test-slave-f892 krb5kdc[10677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:41.927986 10783 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:55:41.929344 10783 webserver.cc:460] Webserver started at http://127.0.97.1:46123/ using document root <none> and password file <none>
I20260927 06:55:41.930064 10783 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:41.930209 10783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:41.930550 10783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:41.933451 10783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "6a451f525a344aacb3a86bfbaa39f469"
format_stamp: "Formatted at 2026-09-27 06:55:41 on dist-test-slave-f892"
server_key: "1aa94d0c187b33c7df39ce72ecf629d6"
server_key_iv: "623315c714951e08c53c02dd4310c711"
server_key_version: "encryptionkey@0"
I20260927 06:55:41.934312 10783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6a451f525a344aacb3a86bfbaa39f469"
format_stamp: "Formatted at 2026-09-27 06:55:41 on dist-test-slave-f892"
server_key: "1aa94d0c187b33c7df39ce72ecf629d6"
server_key_iv: "623315c714951e08c53c02dd4310c711"
server_key_version: "encryptionkey@0"
I20260927 06:55:41.939479 10783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260927 06:55:41.943063 10798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:41.944410 10783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 06:55:41.944569 10783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "6a451f525a344aacb3a86bfbaa39f469"
format_stamp: "Formatted at 2026-09-27 06:55:41 on dist-test-slave-f892"
server_key: "1aa94d0c187b33c7df39ce72ecf629d6"
server_key_iv: "623315c714951e08c53c02dd4310c711"
server_key_version: "encryptionkey@0"
I20260927 06:55:41.944752 10783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:41.964378 10783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:41.968159 10783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:41.968374 10783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:41.969161 10783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:41.970333 10783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:41.970402 10783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:41.970480 10783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:41.970521 10783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:42.001688 10783 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43271
I20260927 06:55:42.001729 10910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43271 every 8 connection(s)
I20260927 06:55:42.002918 10783 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:55:42.011893 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 10783
I20260927 06:55:42.012130 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:55:42.012740 388 external_mini_cluster.cc:1428] Setting key 30836726325119edf513e458c6dc03fc
Sep 27 06:55:42 dist-test-slave-f892 krb5kdc[10677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:55:42.023445 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:42.005355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54571 (local address 127.0.97.62:34697)
0927 06:55:42.005951 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:42.005958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:42.006883 (+ 925us) server_negotiation.cc:407] Connection header received
0927 06:55:42.008593 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:42.008597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:42.008659 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:42.008773 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:42.010112 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:42.010763 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:42.012430 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:42.012821 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:42.012963 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:42.017270 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:42.017275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:42.017279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:42.017310 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:42.019918 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:42.020882 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:42.020885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:42.020888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:42.020998 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:42.021471 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:42.021475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:42.021477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:42.021748 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:42.021853 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:42.022716 (+ 863us) server_negotiation.cc:299] Negotiation successful
0927 06:55:42.022975 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":376,"threads_started":1}
I20260927 06:55:42.024080 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:42.005858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34697 (local address 127.0.97.1:54571)
0927 06:55:42.006731 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:42.006773 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:42.008358 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:42.008960 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:42.008967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:42.009406 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:42.009921 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:42.009937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:42.010917 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:42.010921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:42.012284 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:42.012296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:42.013048 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:42.013056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:42.013345 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:42.014129 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:42.014156 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:42.017021 (+ 2865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:42.020068 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:42.020079 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:42.020100 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:42.020761 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:42.021108 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:42.021115 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:42.021121 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:42.021362 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:42.021878 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:42.021889 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:42.022320 (+ 431us) client_negotiation.cc:769] Sending connection context
0927 06:55:42.022544 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 06:55:42.022945 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":400,"threads_started":1}
I20260927 06:55:42.025703 10911 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34697
I20260927 06:55:42.026031 10911 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:42.026877 10911 heartbeater.cc:511] Master 127.0.97.62:34697 requested a full tablet report, sending...
I20260927 06:55:42.029167 10722 ts_manager.cc:194] Registered new tserver with Master: 6a451f525a344aacb3a86bfbaa39f469 (127.0.97.1:43271)
I20260927 06:55:42.031301 10722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54571
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 06:55:42.064906 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34697
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:55:42.376994 10917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:42.377269 10917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:42.377358 10917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:42.384409 10917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:42.384508 10917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:42.384620 10917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:55:42.393751 10917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:42.394989 10917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:42.396337 10917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:42.410801 10923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:42.410838 10925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:42.411329 10922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:42.412048 10917 server_base.cc:1034] running on GCE node
I20260927 06:55:42.412734 10917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:42.413838 10917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:42.415071 10917 hybrid_clock.cc:648] HybridClock initialized: now 1790492142415007 us; error 53 us; skew 500 ppm
I20260927 06:55:42.415413 10917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:42 dist-test-slave-f892 krb5kdc[10677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:42.419113 10917 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:55:42.420563 10917 webserver.cc:460] Webserver started at http://127.0.97.2:42109/ using document root <none> and password file <none>
I20260927 06:55:42.421269 10917 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:42.421411 10917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:42.421716 10917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:42.424612 10917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "b3e143c870d141cab7abd62b27c7269a"
format_stamp: "Formatted at 2026-09-27 06:55:42 on dist-test-slave-f892"
server_key: "ddbcaeed020181b63e0267538b05d86f"
server_key_iv: "216d5efd0d5d35b4ca4951ac7992e865"
server_key_version: "encryptionkey@0"
I20260927 06:55:42.425355 10917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b3e143c870d141cab7abd62b27c7269a"
format_stamp: "Formatted at 2026-09-27 06:55:42 on dist-test-slave-f892"
server_key: "ddbcaeed020181b63e0267538b05d86f"
server_key_iv: "216d5efd0d5d35b4ca4951ac7992e865"
server_key_version: "encryptionkey@0"
I20260927 06:55:42.430214 10917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260927 06:55:42.433390 10932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:42.434857 10917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:55:42.435016 10917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "b3e143c870d141cab7abd62b27c7269a"
format_stamp: "Formatted at 2026-09-27 06:55:42 on dist-test-slave-f892"
server_key: "ddbcaeed020181b63e0267538b05d86f"
server_key_iv: "216d5efd0d5d35b4ca4951ac7992e865"
server_key_version: "encryptionkey@0"
I20260927 06:55:42.435207 10917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:42.456115 10917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:42.461261 10917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:42.461601 10917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:42.466813 10917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:42.468427 10917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:42.468557 10917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:42.468675 10917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:42.468755 10917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:42.499491 10917 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40391
I20260927 06:55:42.499560 11044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40391 every 8 connection(s)
I20260927 06:55:42.500741 10917 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:55:42.507326 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 10917
I20260927 06:55:42.507524 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:55:42.508085 388 external_mini_cluster.cc:1428] Setting key f79684c7282bab9c14284d79a12ff245
Sep 27 06:55:42 dist-test-slave-f892 krb5kdc[10677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:55:42.521656 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:42.503173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51743 (local address 127.0.97.62:34697)
0927 06:55:42.503346 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:42.503350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:42.504437 (+ 1087us) server_negotiation.cc:407] Connection header received
0927 06:55:42.506165 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:42.506169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:42.506233 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:42.506336 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:42.507762 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:42.508751 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:42.510812 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:42.511204 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:42.511300 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:42.515198 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:42.515203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:42.515206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:42.515235 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:42.518412 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:42.519338 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:42.519342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:42.519344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:42.519479 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:42.519970 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:42.519974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:42.519976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:42.520256 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:42.520364 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:42.521245 (+ 881us) server_negotiation.cc:299] Negotiation successful
0927 06:55:42.521411 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 06:55:42.523118 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:42.503607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34697 (local address 127.0.97.2:51743)
0927 06:55:42.504260 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:42.504298 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:42.505863 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:42.506547 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:42.506554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:42.506997 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:42.507577 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:42.507602 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:42.508915 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:42.508920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:42.510672 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:42.510688 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:42.511429 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:42.511438 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:42.511758 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:42.512468 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:42.512501 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:42.514994 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:42.518559 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:42.518570 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:42.518589 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:42.519192 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:42.519592 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:42.519599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:42.519606 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:42.519841 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:42.520381 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:42.520391 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:42.520832 (+ 441us) client_negotiation.cc:769] Sending connection context
0927 06:55:42.521140 (+ 308us) client_negotiation.cc:240] Negotiation successful
0927 06:55:42.521687 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":255,"threads_started":1}
I20260927 06:55:42.524549 11045 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34697
I20260927 06:55:42.524879 11045 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:42.525661 11045 heartbeater.cc:511] Master 127.0.97.62:34697 requested a full tablet report, sending...
I20260927 06:55:42.527158 10722 ts_manager.cc:194] Registered new tserver with Master: b3e143c870d141cab7abd62b27c7269a (127.0.97.2:40391)
I20260927 06:55:42.527869 10722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51743
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 06:55:42.561322 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34697
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:55:42.875011 11050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:55:42.875401 11050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:55:42.875599 11050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:55:42.886322 11050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:55:42.886454 11050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:55:42.886639 11050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:55:42.896888 11050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:55:42.898243 11050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:55:42.899551 11050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:55:42.912052 11055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:42.912348 11056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:55:42.912354 11058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:55:42.913234 11050 server_base.cc:1034] running on GCE node
I20260927 06:55:42.913771 11050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:55:42.914626 11050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:55:42.915844 11050 hybrid_clock.cc:648] HybridClock initialized: now 1790492142915799 us; error 36 us; skew 500 ppm
I20260927 06:55:42.916167 11050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:55:42 dist-test-slave-f892 krb5kdc[10677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:55:42.919318 11050 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:55:42.920761 11050 webserver.cc:460] Webserver started at http://127.0.97.3:39245/ using document root <none> and password file <none>
I20260927 06:55:42.921440 11050 fs_manager.cc:359] Metadata directory not provided
I20260927 06:55:42.921576 11050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:55:42.921861 11050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:55:42.924669 11050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "cdee837da286422898b1c1efeafba060"
format_stamp: "Formatted at 2026-09-27 06:55:42 on dist-test-slave-f892"
server_key: "63097b12bcd7ea5e1c7b8adb97562df2"
server_key_iv: "6fcda1e58de3055edc6878680502eb15"
server_key_version: "encryptionkey@0"
I20260927 06:55:42.925382 11050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cdee837da286422898b1c1efeafba060"
format_stamp: "Formatted at 2026-09-27 06:55:42 on dist-test-slave-f892"
server_key: "63097b12bcd7ea5e1c7b8adb97562df2"
server_key_iv: "6fcda1e58de3055edc6878680502eb15"
server_key_version: "encryptionkey@0"
I20260927 06:55:42.930339 11050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 06:55:42.933579 11065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:42.934968 11050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:55:42.935163 11050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "cdee837da286422898b1c1efeafba060"
format_stamp: "Formatted at 2026-09-27 06:55:42 on dist-test-slave-f892"
server_key: "63097b12bcd7ea5e1c7b8adb97562df2"
server_key_iv: "6fcda1e58de3055edc6878680502eb15"
server_key_version: "encryptionkey@0"
I20260927 06:55:42.935357 11050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:55:42.958664 11050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:55:42.962468 11050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:55:42.962741 11050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:55:42.963649 11050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:55:42.964879 11050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:55:42.964978 11050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:42.965127 11050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:55:42.965198 11050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:55:42.998524 11050 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41823
I20260927 06:55:42.998579 11177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41823 every 8 connection(s)
I20260927 06:55:42.999787 11050 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:55:43.001214 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 11050
I20260927 06:55:43.001399 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:55:43.001886 388 external_mini_cluster.cc:1428] Setting key 4923513896fdc0743651a0f1bd7c07d8
Sep 27 06:55:43 dist-test-slave-f892 krb5kdc[10677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:55:43.019532 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:43.002521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47497 (local address 127.0.97.62:34697)
0927 06:55:43.002784 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:43.002790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:43.004289 (+ 1499us) server_negotiation.cc:407] Connection header received
0927 06:55:43.006260 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:43.006266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:43.006342 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:43.006452 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:43.007724 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.008505 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.009964 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.010448 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.010564 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:43.013738 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:43.013743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:43.013746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:43.013777 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:43.016603 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:43.017491 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:43.017496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:43.017498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:43.017629 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:43.018072 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:43.018076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:43.018079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:43.018345 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:43.018458 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:43.019116 (+ 658us) server_negotiation.cc:299] Negotiation successful
0927 06:55:43.019281 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260927 06:55:43.020491 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:43.003004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34697 (local address 127.0.97.3:47497)
0927 06:55:43.004146 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:43.004190 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:43.005955 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:43.006644 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:43.006652 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:43.007083 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:55:43.007569 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:43.007584 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.008665 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.008668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:43.009828 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:43.009838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.010669 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.010674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:43.010848 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:43.011494 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:55:43.011514 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:55:43.013543 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:55:43.016755 (+ 3212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:43.016793 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:43.016811 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:43.017375 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:43.017749 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:55:43.017752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:55:43.017755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:55:43.017929 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:55:43.018464 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:55:43.018478 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:55:43.018799 (+ 321us) client_negotiation.cc:769] Sending connection context
0927 06:55:43.019028 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 06:55:43.019422 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"thread_start_us":620,"threads_started":1}
I20260927 06:55:43.021906 11178 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34697
I20260927 06:55:43.022291 11178 heartbeater.cc:464] Registering TS with master...
I20260927 06:55:43.022984 11178 heartbeater.cc:511] Master 127.0.97.62:34697 requested a full tablet report, sending...
I20260927 06:55:43.024379 10722 ts_manager.cc:194] Registered new tserver with Master: cdee837da286422898b1c1efeafba060 (127.0.97.3:41823)
I20260927 06:55:43.025192 10722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47497
I20260927 06:55:43.029150 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:55:43.034940 10911 heartbeater.cc:503] Master 127.0.97.62:34697 was elected leader, sending a full tablet report...
I20260927 06:55:43.045591 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:43.033296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43482 (local address 127.0.97.62:34697)
0927 06:55:43.033444 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:43.033448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:43.033900 (+ 452us) server_negotiation.cc:407] Connection header received
0927 06:55:43.034077 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:43.034080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:43.034140 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:43.034215 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:43.035523 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.036398 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.037564 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.037920 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.038120 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:43.040423 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:43.040429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:43.040433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:43.040470 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:43.043026 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:43.043651 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:43.043657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:43.043661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:43.043775 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:43.044168 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:43.044175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:43.044178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:43.044452 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:43.044678 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:43.045025 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 06:55:43.045290 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 06:55:43.049132 10722 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:55:43.055840 10722 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:55:43.078562 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:43.067289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40391 (local address 127.0.0.1:55660)
0927 06:55:43.068334 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:43.068391 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:43.068612 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:43.070533 (+ 1921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:43.070541 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:43.070606 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:43.071018 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:43.071032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.073196 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.073201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:43.075724 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:43.075744 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.077588 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.077594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:43.077804 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:43.077857 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 06:55:43.077952 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 06:55:43.078144 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":4,"spinlock_wait_cycles":15744,"thread_start_us":8312,"threads_started":1}
I20260927 06:55:43.078934 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:43.067448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55660 (local address 127.0.97.2:40391)
0927 06:55:43.069883 (+ 2435us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:43.069892 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:43.069918 (+ 26us) server_negotiation.cc:407] Connection header received
0927 06:55:43.070008 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:43.070062 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:43.070311 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:43.071201 (+ 890us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:43.071423 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.073042 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.075938 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.077375 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.078109 (+ 734us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:43.078339 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:43.078408 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 06:55:43.078590 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2303,"thread_start_us":2182,"threads_started":1}
I20260927 06:55:43.079674 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:43.067383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41823 (local address 127.0.0.1:37260)
0927 06:55:43.070382 (+ 2999us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:43.070407 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:43.070577 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:43.071447 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:43.071451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:43.071479 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:43.071885 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:43.071896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.073818 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.073823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:43.076534 (+ 2711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:43.076547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.078742 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.078749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:43.078945 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:43.078989 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 06:55:43.079124 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 06:55:43.079265 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2877,"mutex_wait_us":634,"thread_start_us":886,"threads_started":1}
I20260927 06:55:43.079850 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:43.068074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37260 (local address 127.0.97.3:41823)
0927 06:55:43.069460 (+ 1386us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:43.069473 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:43.070487 (+ 1014us) server_negotiation.cc:407] Connection header received
0927 06:55:43.070760 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:43.070768 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:43.070985 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:43.071161 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:43.072130 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.073647 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.076713 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.078260 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.078390 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:43.078597 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:43.079285 (+ 688us) server_negotiation.cc:299] Negotiation successful
0927 06:55:43.079470 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1178,"thread_start_us":1756,"threads_started":1}
I20260927 06:55:43.082548 10980 tablet_service.cc:1467] Processing CreateTablet for tablet 82f76a2c387f4e48b6e4c34535142a58 (DEFAULT_TABLE table=test-table [id=76f7512b64d7489a93b226e9d9dd0b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:43.082883 11113 tablet_service.cc:1467] Processing CreateTablet for tablet 82f76a2c387f4e48b6e4c34535142a58 (DEFAULT_TABLE table=test-table [id=76f7512b64d7489a93b226e9d9dd0b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:43.083950 10980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f76a2c387f4e48b6e4c34535142a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:43.084071 11113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f76a2c387f4e48b6e4c34535142a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:43.084549 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:43.069205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43271 (local address 127.0.0.1:47408)
0927 06:55:43.075881 (+ 6676us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:43.075906 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:43.076002 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:43.076816 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:43.076821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:43.076856 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:43.077152 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:43.077162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.079001 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.079007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:43.081726 (+ 2719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:43.081740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.083843 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.083849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:43.084019 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:43.084048 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 06:55:43.084135 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 06:55:43.084250 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6537,"thread_start_us":924,"threads_started":1}
I20260927 06:55:43.085214 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:43.069909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47408 (local address 127.0.97.1:43271)
0927 06:55:43.076318 (+ 6409us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:43.076325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:43.076348 (+ 23us) server_negotiation.cc:407] Connection header received
0927 06:55:43.076423 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:43.076428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:43.076586 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:43.076842 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:43.077326 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.078843 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.081903 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.083380 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.083499 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:43.083718 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:43.084749 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0927 06:55:43.084910 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6261,"thread_start_us":6111,"threads_started":1}
I20260927 06:55:43.088181 10846 tablet_service.cc:1467] Processing CreateTablet for tablet 82f76a2c387f4e48b6e4c34535142a58 (DEFAULT_TABLE table=test-table [id=76f7512b64d7489a93b226e9d9dd0b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:55:43.089354 10846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f76a2c387f4e48b6e4c34535142a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:55:43.096200 11196 tablet_bootstrap.cc:492] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060: Bootstrap starting.
I20260927 06:55:43.100548 11196 tablet_bootstrap.cc:654] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:43.101051 11197 tablet_bootstrap.cc:492] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a: Bootstrap starting.
I20260927 06:55:43.102109 11196 log.cc:824] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060: Log is configured to *not* fsync() on all Append() calls
I20260927 06:55:43.104529 11199 tablet_bootstrap.cc:492] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469: Bootstrap starting.
I20260927 06:55:43.104808 11197 tablet_bootstrap.cc:654] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:43.105499 11196 tablet_bootstrap.cc:492] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060: No bootstrap required, opened a new log
I20260927 06:55:43.105780 11196 ts_tablet_manager.cc:1397] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 06:55:43.106138 11197 log.cc:824] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a: Log is configured to *not* fsync() on all Append() calls
I20260927 06:55:43.107399 11199 tablet_bootstrap.cc:654] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469: Neither blocks nor log segments found. Creating new log.
I20260927 06:55:43.108474 11199 log.cc:824] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469: Log is configured to *not* fsync() on all Append() calls
I20260927 06:55:43.112488 11197 tablet_bootstrap.cc:492] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a: No bootstrap required, opened a new log
I20260927 06:55:43.112618 11199 tablet_bootstrap.cc:492] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469: No bootstrap required, opened a new log
I20260927 06:55:43.112756 11197 ts_tablet_manager.cc:1397] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260927 06:55:43.112828 11199 ts_tablet_manager.cc:1397] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260927 06:55:43.115981 11196 raft_consensus.cc:348] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e143c870d141cab7abd62b27c7269a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "cdee837da286422898b1c1efeafba060" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41823 } } peers { permanent_uuid: "6a451f525a344aacb3a86bfbaa39f469" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43271 } }
I20260927 06:55:43.116364 11196 raft_consensus.cc:374] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:43.116510 11196 raft_consensus.cc:729] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdee837da286422898b1c1efeafba060, State: Initialized, Role: FOLLOWER
I20260927 06:55:43.117261 11196 consensus_queue.cc:260] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e143c870d141cab7abd62b27c7269a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "cdee837da286422898b1c1efeafba060" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41823 } } peers { permanent_uuid: "6a451f525a344aacb3a86bfbaa39f469" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43271 } }
I20260927 06:55:43.118400 11178 heartbeater.cc:503] Master 127.0.97.62:34697 was elected leader, sending a full tablet report...
I20260927 06:55:43.119820 11196 ts_tablet_manager.cc:1428] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260927 06:55:43.123184 11197 raft_consensus.cc:348] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e143c870d141cab7abd62b27c7269a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "cdee837da286422898b1c1efeafba060" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41823 } } peers { permanent_uuid: "6a451f525a344aacb3a86bfbaa39f469" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43271 } }
I20260927 06:55:43.123621 11197 raft_consensus.cc:374] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:43.123723 11197 raft_consensus.cc:729] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e143c870d141cab7abd62b27c7269a, State: Initialized, Role: FOLLOWER
I20260927 06:55:43.123785 11199 raft_consensus.cc:348] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e143c870d141cab7abd62b27c7269a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "cdee837da286422898b1c1efeafba060" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41823 } } peers { permanent_uuid: "6a451f525a344aacb3a86bfbaa39f469" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43271 } }
I20260927 06:55:43.124176 11199 raft_consensus.cc:374] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:55:43.124284 11199 raft_consensus.cc:729] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a451f525a344aacb3a86bfbaa39f469, State: Initialized, Role: FOLLOWER
I20260927 06:55:43.124420 11197 consensus_queue.cc:260] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e143c870d141cab7abd62b27c7269a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "cdee837da286422898b1c1efeafba060" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41823 } } peers { permanent_uuid: "6a451f525a344aacb3a86bfbaa39f469" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43271 } }
I20260927 06:55:43.124830 11199 consensus_queue.cc:260] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e143c870d141cab7abd62b27c7269a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "cdee837da286422898b1c1efeafba060" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41823 } } peers { permanent_uuid: "6a451f525a344aacb3a86bfbaa39f469" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43271 } }
I20260927 06:55:43.125535 11045 heartbeater.cc:503] Master 127.0.97.62:34697 was elected leader, sending a full tablet report...
I20260927 06:55:43.126178 11197 ts_tablet_manager.cc:1428] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260927 06:55:43.126557 11199 ts_tablet_manager.cc:1428] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
W20260927 06:55:43.251353 11179 tablet.cc:2369] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:55:43.252678 11046 tablet.cc:2369] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:55:43.255102 10912 tablet.cc:2369] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:55:43.534385 11203 raft_consensus.cc:482] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:55:43.534698 11203 raft_consensus.cc:504] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e143c870d141cab7abd62b27c7269a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "cdee837da286422898b1c1efeafba060" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41823 } } peers { permanent_uuid: "6a451f525a344aacb3a86bfbaa39f469" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43271 } }
I20260927 06:55:43.536398 11203 leader_election.cc:288] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdee837da286422898b1c1efeafba060 (127.0.97.3:41823), 6a451f525a344aacb3a86bfbaa39f469 (127.0.97.1:43271)
I20260927 06:55:43.543576 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:43.536789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58007 (local address 127.0.97.3:41823)
0927 06:55:43.536954 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:43.536958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:43.537540 (+ 582us) server_negotiation.cc:407] Connection header received
0927 06:55:43.537638 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:43.537643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:43.537716 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:43.537820 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:43.538574 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.539566 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.541595 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.542667 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.542757 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:43.542827 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:43.543215 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 06:55:43.543309 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 06:55:43.543763 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:43.536713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41823 (local address 127.0.97.2:58007)
0927 06:55:43.537317 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:43.537341 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:43.537475 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:43.538038 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:43.538048 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:43.538075 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:43.538433 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:43.538444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.539739 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.539745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:43.541452 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:43.541464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.542860 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.542866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:43.543035 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:43.543083 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 06:55:43.543186 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 06:55:43.543314 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":1,"thread_start_us":320,"threads_started":1}
I20260927 06:55:43.544364 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:43.537470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43271 (local address 127.0.97.2:50751)
0927 06:55:43.538293 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 06:55:43.538315 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 06:55:43.538454 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:55:43.538787 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:55:43.538790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:55:43.538808 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:55:43.539127 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:43.539134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.540395 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.540398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:43.542185 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:55:43.542198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.543686 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.543695 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:55:43.543890 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:43.543913 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 06:55:43.544030 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 06:55:43.544158 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":415,"threads_started":1}
I20260927 06:55:43.544796 11133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f76a2c387f4e48b6e4c34535142a58" candidate_uuid: "b3e143c870d141cab7abd62b27c7269a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdee837da286422898b1c1efeafba060" is_pre_election: true
I20260927 06:55:43.545310 11133 raft_consensus.cc:2454] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3e143c870d141cab7abd62b27c7269a in term 0.
I20260927 06:55:43.545825 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:43.537681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50751 (local address 127.0.97.1:43271)
0927 06:55:43.537929 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:43.537934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:43.538428 (+ 494us) server_negotiation.cc:407] Connection header received
0927 06:55:43.538594 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:43.538598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:43.538675 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:43.538769 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:55:43.539270 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.540266 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.542323 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.543430 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.543553 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:43.543669 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:43.544101 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 06:55:43.544224 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 06:55:43.545943 10933 leader_election.cc:302] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3e143c870d141cab7abd62b27c7269a, cdee837da286422898b1c1efeafba060; no voters:
I20260927 06:55:43.546360 11203 raft_consensus.cc:2781] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:55:43.546492 11203 raft_consensus.cc:482] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:55:43.546617 11203 raft_consensus.cc:3037] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:43.546775 10866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f76a2c387f4e48b6e4c34535142a58" candidate_uuid: "b3e143c870d141cab7abd62b27c7269a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a451f525a344aacb3a86bfbaa39f469" is_pre_election: true
I20260927 06:55:43.547204 10866 raft_consensus.cc:2454] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3e143c870d141cab7abd62b27c7269a in term 0.
I20260927 06:55:43.549072 11203 raft_consensus.cc:504] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e143c870d141cab7abd62b27c7269a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "cdee837da286422898b1c1efeafba060" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41823 } } peers { permanent_uuid: "6a451f525a344aacb3a86bfbaa39f469" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43271 } }
I20260927 06:55:43.549667 11203 leader_election.cc:288] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [CANDIDATE]: Term 1 election: Requested vote from peers cdee837da286422898b1c1efeafba060 (127.0.97.3:41823), 6a451f525a344aacb3a86bfbaa39f469 (127.0.97.1:43271)
I20260927 06:55:43.550109 10866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f76a2c387f4e48b6e4c34535142a58" candidate_uuid: "b3e143c870d141cab7abd62b27c7269a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a451f525a344aacb3a86bfbaa39f469"
I20260927 06:55:43.550099 11133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f76a2c387f4e48b6e4c34535142a58" candidate_uuid: "b3e143c870d141cab7abd62b27c7269a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdee837da286422898b1c1efeafba060"
I20260927 06:55:43.550278 10866 raft_consensus.cc:3037] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:43.550297 11133 raft_consensus.cc:3037] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:55:43.552312 10866 raft_consensus.cc:2454] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e143c870d141cab7abd62b27c7269a in term 1.
I20260927 06:55:43.552416 11133 raft_consensus.cc:2454] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e143c870d141cab7abd62b27c7269a in term 1.
I20260927 06:55:43.552716 10933 leader_election.cc:302] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a451f525a344aacb3a86bfbaa39f469, b3e143c870d141cab7abd62b27c7269a; no voters:
I20260927 06:55:43.552989 11203 raft_consensus.cc:2781] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:55:43.553395 11203 raft_consensus.cc:686] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [term 1 LEADER]: Becoming Leader. State: Replica: b3e143c870d141cab7abd62b27c7269a, State: Running, Role: LEADER
I20260927 06:55:43.553819 11203 consensus_queue.cc:237] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e143c870d141cab7abd62b27c7269a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "cdee837da286422898b1c1efeafba060" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41823 } } peers { permanent_uuid: "6a451f525a344aacb3a86bfbaa39f469" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43271 } }
I20260927 06:55:43.557993 10722 catalog_manager.cc:5495] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a reported cstate change: term changed from 0 to 1, leader changed from <none> to b3e143c870d141cab7abd62b27c7269a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b3e143c870d141cab7abd62b27c7269a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e143c870d141cab7abd62b27c7269a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdee837da286422898b1c1efeafba060" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a451f525a344aacb3a86bfbaa39f469" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20260927 06:55:43.642771 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:43.636778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55674 (local address 127.0.97.2:40391)
0927 06:55:43.637423 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:43.637429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:43.637448 (+ 19us) server_negotiation.cc:407] Connection header received
0927 06:55:43.637590 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:43.637594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:43.637672 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:43.637832 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:55:43.638299 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.639096 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.640560 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.640952 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.641081 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:43.641616 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:55:43.642197 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:55:43.642370 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:43.642418 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 06:55:43.642498 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":434,"threads_started":1}
I20260927 06:55:43.650218 11133 raft_consensus.cc:1261] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060 [term 1 FOLLOWER]: Refusing update from remote peer b3e143c870d141cab7abd62b27c7269a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:55:43.650218 10866 raft_consensus.cc:1261] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469 [term 1 FOLLOWER]: Refusing update from remote peer b3e143c870d141cab7abd62b27c7269a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:55:43.651166 11207 consensus_queue.cc:1035] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a451f525a344aacb3a86bfbaa39f469" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:55:43.651495 11203 consensus_queue.cc:1035] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdee837da286422898b1c1efeafba060" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:55:43.668496 11216 mvcc.cc:205] Tried to move back new op lower bound from 7333855820381114368 to 7333855819999657984. Current Snapshot: MvccSnapshot[applied={T|T < 7333855820381114368}]
I20260927 06:55:43.674947 10721 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:55:43.675244 10721 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:55:43.678717 10721 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d97c6585f5864c09a9a61338d631f3a8: Sending DeleteTablet for 3 replicas of tablet 82f76a2c387f4e48b6e4c34535142a58
I20260927 06:55:43.679828 10980 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f76a2c387f4e48b6e4c34535142a58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55660
I20260927 06:55:43.679937 10846 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f76a2c387f4e48b6e4c34535142a58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47408
I20260927 06:55:43.679961 11113 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f76a2c387f4e48b6e4c34535142a58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:55:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37260
I20260927 06:55:43.680737 11224 tablet_replica.cc:330] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a: stopping tablet replica
I20260927 06:55:43.681195 11224 raft_consensus.cc:2229] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:55:43.681851 11224 raft_consensus.cc:2258] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:43.682070 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 10783
I20260927 06:55:43.682471 11225 tablet_replica.cc:330] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469: stopping tablet replica
I20260927 06:55:43.682801 11225 raft_consensus.cc:2229] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:55:43.683213 11225 raft_consensus.cc:2258] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:43.684794 11225 ts_tablet_manager.cc:1905] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:43.691210 11224 ts_tablet_manager.cc:1905] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:55:43.698647 11224 ts_tablet_manager.cc:1918] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:55:43.698823 11224 log.cc:1196] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/82f76a2c387f4e48b6e4c34535142a58
I20260927 06:55:43.699362 11224 ts_tablet_manager.cc:1939] T 82f76a2c387f4e48b6e4c34535142a58 P b3e143c870d141cab7abd62b27c7269a: Deleting consensus metadata
I20260927 06:55:43.701277 10709 catalog_manager.cc:4841] TS b3e143c870d141cab7abd62b27c7269a (127.0.97.2:40391): tablet 82f76a2c387f4e48b6e4c34535142a58 (table test-table [id=76f7512b64d7489a93b226e9d9dd0b65]) successfully deleted
I20260927 06:55:43.702744 11225 ts_tablet_manager.cc:1918] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:55:43.702953 11225 log.cc:1196] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/82f76a2c387f4e48b6e4c34535142a58
I20260927 06:55:43.703503 11225 ts_tablet_manager.cc:1939] T 82f76a2c387f4e48b6e4c34535142a58 P 6a451f525a344aacb3a86bfbaa39f469: Deleting consensus metadata
I20260927 06:55:43.705221 10706 catalog_manager.cc:4841] TS 6a451f525a344aacb3a86bfbaa39f469 (127.0.97.1:43271): tablet 82f76a2c387f4e48b6e4c34535142a58 (table test-table [id=76f7512b64d7489a93b226e9d9dd0b65]) successfully deleted
I20260927 06:55:43.706142 11226 tablet_replica.cc:330] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060: stopping tablet replica
I20260927 06:55:43.706522 11226 raft_consensus.cc:2229] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:55:43.706955 11226 raft_consensus.cc:2258] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:55:43.708803 11226 ts_tablet_manager.cc:1905] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 06:55:43 dist-test-slave-f892 krb5kdc[10677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:55:43.716050 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:43.685138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47416 (local address 127.0.97.1:43271)
0927 06:55:43.685350 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:43.685359 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:43.694593 (+ 9234us) server_negotiation.cc:407] Connection header received
0927 06:55:43.695220 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:43.695230 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:43.695342 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:43.695442 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:43.697704 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.699417 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.703068 (+ 3651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.703982 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.704124 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:43.710674 (+ 6550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:43.710685 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:43.710698 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:43.710730 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:43.713283 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:43.713933 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:43.713940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:43.713942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:43.714132 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:43.714501 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:43.714506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:43.714509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:43.714828 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:43.715122 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:43.715408 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 06:55:43.715625 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 06:55:43.716140 11226 ts_tablet_manager.cc:1918] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:55:43.716275 11226 log.cc:1196] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/82f76a2c387f4e48b6e4c34535142a58
I20260927 06:55:43.716725 10906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:43.716784 11226 ts_tablet_manager.cc:1939] T 82f76a2c387f4e48b6e4c34535142a58 P cdee837da286422898b1c1efeafba060: Deleting consensus metadata
I20260927 06:55:43.718627 10706 catalog_manager.cc:4841] TS cdee837da286422898b1c1efeafba060 (127.0.97.3:41823): tablet 82f76a2c387f4e48b6e4c34535142a58 (table test-table [id=76f7512b64d7489a93b226e9d9dd0b65]) successfully deleted
I20260927 06:55:43.825738 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 10783
I20260927 06:55:43.844185 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 10917
Sep 27 06:55:43 dist-test-slave-f892 krb5kdc[10677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:55:43.861029 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:43.844955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55688 (local address 127.0.97.2:40391)
0927 06:55:43.845141 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:43.845147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:43.845168 (+ 21us) server_negotiation.cc:407] Connection header received
0927 06:55:43.845347 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:43.845353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:43.845436 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:43.845537 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:43.846687 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.847798 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.849014 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.849699 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.849847 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:43.854220 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:43.854238 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:43.854258 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:43.854300 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:43.858335 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:43.858945 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:43.858954 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:43.858958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:43.859115 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:43.859431 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:43.859436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:43.859440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:43.859861 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:43.860155 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:43.860376 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 06:55:43.860640 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 06:55:43.861783 11040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:43.972647 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 10917
I20260927 06:55:43.990792 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 11050
Sep 27 06:55:43 dist-test-slave-f892 krb5kdc[10677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 06:55:44.006100 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:55:43.991534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.0.97.3:41823)
0927 06:55:43.991729 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 06:55:43.991735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:55:43.991755 (+ 20us) server_negotiation.cc:407] Connection header received
0927 06:55:43.991947 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:55:43.991956 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:55:43.992057 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:55:43.992180 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:55:43.993363 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.994172 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.995404 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:55:43.995796 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:55:43.995924 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:55:44.000589 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:55:44.000599 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:55:44.000610 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:55:44.000639 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:55:44.003425 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:44.004052 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:44.004059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:44.004062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:44.004179 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:55:44.004544 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:55:44.004555 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:55:44.004558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:55:44.004913 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:55:44.005187 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 06:55:44.005512 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 06:55:44.005724 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 06:55:44.006712 11173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:44.091775 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 11050
I20260927 06:55:44.110435 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 10690
I20260927 06:55:44.111568 10752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:55:44.216969 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 10690
2026-09-27T06:55:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4574 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:55:44 dist-test-slave-f892 krb5kdc[11234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:55:44 dist-test-slave-f892 krb5kdc[11234](info): set up 2 sockets
Sep 27 06:55:44 dist-test-slave-f892 krb5kdc[11234](info): commencing operation
krb5kdc: starting...
W20260927 06:55:45.314831 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:55:45 dist-test-slave-f892 krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:55:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:55:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260927 06:55:45.634644 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasktaM5HI/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/postgres -l logfile start
2026-09-27 06:55:48.065 UTC [11264] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 06:55:48.065 UTC [11264] LOG: listening on IPv4 address "127.0.97.20", port 52203
2026-09-27 06:55:48.068 UTC [11264] LOG: listening on Unix socket "/tmp/.s.PGSQL.52203"
2026-09-27 06:55:48.091 UTC [11267] LOG: database system was shut down at 2026-09-27 06:55:46 UTC
2026-09-27 06:55:48.094 UTC [11264] LOG: database system is ready to accept connections
I20260927 06:55:49.074532 388 mini_postgres.cc:96] Postgres bound to 52203
2026-09-27 06:55:49.091 UTC [11275] FATAL: database "slave" does not exist
127.0.97.20:52203 - accepting connections
I20260927 06:55:49.091995 388 mini_ranger.cc:161] Starting Ranger...
I20260927 06:55:49.116070 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 06:55:49.579209 388 mini_ranger.cc:87] Created ranger Postgres database
I20260927 06:55:49.579345 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ranger-admin
2026-09-27 06:55:49,867 [I] DB FLAVOR :POSTGRES
2026-09-27 06:55:49,868 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 06:55:49,868 [I] Checking connection..
2026-09-27 06:55:49,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 06:55:50,116 [I] Checking connection passed.
2026-09-27 06:55:50,117 [I] --------- Verifying version history table ---------
2026-09-27 06:55:50,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 06:55:50,356 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 06:55:50,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 06:55:50,605 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 06:55:50,606 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 06:55:50,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 06:55:50.824 UTC [11339] WARNING: there is no transaction in progress
2026-09-27 06:55:50,830 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 06:55:50,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 06:55:51,069 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 06:55:51,069 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 06:55:51,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 06:55:51,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 06:55:51,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f892.c.gcp-upstream.internal','N') ;"
2026-09-27 06:55:51,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 06:55:52,004 [I] Table x_portal_user does not exist in database ranger
2026-09-27 06:55:52,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 06:55:52,257 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 06:55:52,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 06:55:52,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 06:55:52,747 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 06:55:52,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 06:55:53.663 UTC [11447] WARNING: there is no transaction in progress
2026-09-27 06:55:53.679 UTC [11447] WARNING: there is no transaction in progress
2026-09-27 06:55:53.695 UTC [11447] WARNING: there is no transaction in progress
2026-09-27 06:55:53.709 UTC [11447] WARNING: there is no transaction in progress
2026-09-27 06:55:53.725 UTC [11447] WARNING: there is no transaction in progress
2026-09-27 06:55:53.741 UTC [11447] WARNING: there is no transaction in progress
2026-09-27 06:55:53.793 UTC [11447] WARNING: there is no transaction in progress
2026-09-27 06:55:53.800 UTC [11447] WARNING: there is no transaction in progress
2026-09-27 06:55:53.813 UTC [11447] WARNING: there is no transaction in progress
2026-09-27 06:55:53.823 UTC [11447] WARNING: there is no transaction in progress
2026-09-27 06:55:53.833 UTC [11447] WARNING: there is no transaction in progress
2026-09-27 06:55:53.845 UTC [11447] WARNING: there is no transaction in progress
2026-09-27 06:55:53.854 UTC [11447] WARNING: there is no transaction in progress
2026-09-27 06:55:53.860 UTC [11447] WARNING: there is no transaction in progress
2026-09-27 06:55:53.866 UTC [11447] WARNING: there is no transaction in progress
2026-09-27 06:55:53.872 UTC [11447] WARNING: there is no transaction in progress
2026-09-27 06:55:54,466 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 06:55:54,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 06:55:54,698 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 06:55:54,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 06:55:54,940 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 06:55:54,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 06:55:55,186 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 06:55:55,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 06:55:55,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 06:55:55,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f892.c.gcp-upstream.internal';"
2026-09-27 06:55:55,890 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 06:55:55,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 06:55:56,141 [I] DB_PATCHES have already been applied
I20260927 06:55:56.147421 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ranger-admin:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasktaM5HI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 06:55:56.147501 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260927 06:55:56.161731 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:58887
Sep 27, 2026 6:55:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:55:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:55:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 6:55:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 6:55:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 6:55:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 6:55:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 06:55:57.299863 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.208s user 0.000s sys 0.035s
Sep 27, 2026 6:56:16 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 6:56:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 6:56:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 6:56:16 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 6:56:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 6:56:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 6:56:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 6:56:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 6:56:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 6:56:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 6:56:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 6:56:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 6:56:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 6:56:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 6:56:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 6:56:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 6:56:16 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 6:56:17 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 06:56:17.740909 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 06:56:17 dist-test-slave-f892 krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492177, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 06:56:17.916738 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 06:56:17.936120 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 06:56:18.756237 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 06:56:18.756408 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ranger-kms
I20260927 06:56:18.756557 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:52203
2026-09-27 06:56:19,163 [I] DB FLAVOR :POSTGRES
2026-09-27 06:56:19,163 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 06:56:19,164 [I] Checking connection
2026-09-27 06:56:19,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 06:56:19,431 [I] connection success
2026-09-27 06:56:19,431 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 06:56:19,431 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 06:56:19,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 06:56:19,684 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 06:56:19,685 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 06:56:19,685 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 06:56:19,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:52203/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 06:56:19,967 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 06:56:20.416981 388 mini_ranger_kms.cc:310] Created kms service
I20260927 06:56:20.587656 388 mini_ranger_kms.cc:326] Created kudu user
I20260927 06:56:20.631937 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 06:56:20 dist-test-slave-f892 krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492177, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 06:56:21.040552 388 mini_ranger_kms.cc:384] Added ranger policy
I20260927 06:56:21.040874 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ranger-kms:/tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 06:56:21.040938 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260927 06:56:21.061559 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11627
I20260927 06:56:21.061740 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:51399
Sep 27, 2026 6:56:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:56:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:56:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 6:56:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 6:56:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 6:56:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 06:56:23.350344 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.434s user 0.124s sys 0.296s
I20260927 06:56:23.350596 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 06:56:23.350812 388 mini_ranger_kms.cc:401] 127.0.97.20:51399/kms/v1/keys
Sep 27 06:56:23 dist-test-slave-f892 krb5kdc[11234](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492183, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 06:56:23 dist-test-slave-f892 krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492183, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 06:56:24 dist-test-slave-f892 krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492183, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 06:56:26 dist-test-slave-f892 krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:56:26.570528 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45147
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45147
--ranger_config_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51399/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:56:26.901779 11678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:56:26.902123 11678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:56:26.902282 11678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:56:26.909548 11678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:56:26.909693 11678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:56:26.909790 11678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:56:26.909904 11678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:56:26.909982 11678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:56:26.920132 11678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:51399/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45147
--ranger_config_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:56:26.921367 11678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:56:26.922691 11678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:56:26.936496 11684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:56:26.936563 11686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:56:26.936988 11683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:56:26.937091 11678 server_base.cc:1034] running on GCE node
I20260927 06:56:26.937971 11678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:56:26.939630 11678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:56:26.940872 11678 hybrid_clock.cc:648] HybridClock initialized: now 1790492186940801 us; error 85 us; skew 500 ppm
I20260927 06:56:26.941251 11678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:56:26 dist-test-slave-f892 krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:56:26.949541 11678 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:56:26.951380 11678 webserver.cc:460] Webserver started at http://127.0.97.62:41677/ using document root <none> and password file <none>
I20260927 06:56:26.952020 11678 fs_manager.cc:359] Metadata directory not provided
I20260927 06:56:26.952157 11678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:56:26.952455 11678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 06:56:26 dist-test-slave-f892 krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 06:56:27.007838 11678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "5c92fdaf6a38425db931ec95d99b45c8"
format_stamp: "Formatted at 2026-09-27 06:56:26 on dist-test-slave-f892"
server_key: "639012e33a84d0fec9435e32a46e4c77"
server_key_iv: "5d2d315a6e93aee54f4b4ea4dda15a7d"
server_key_version: "kuduclusterkey@0"
I20260927 06:56:27.008724 11678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "5c92fdaf6a38425db931ec95d99b45c8"
format_stamp: "Formatted at 2026-09-27 06:56:26 on dist-test-slave-f892"
server_key: "639012e33a84d0fec9435e32a46e4c77"
server_key_iv: "5d2d315a6e93aee54f4b4ea4dda15a7d"
server_key_version: "kuduclusterkey@0"
I20260927 06:56:27.013991 11678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 06:56:27.033552 11693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:56:27.035475 11678 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 06:56:27.035684 11678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "5c92fdaf6a38425db931ec95d99b45c8"
format_stamp: "Formatted at 2026-09-27 06:56:26 on dist-test-slave-f892"
server_key: "639012e33a84d0fec9435e32a46e4c77"
server_key_iv: "5d2d315a6e93aee54f4b4ea4dda15a7d"
server_key_version: "kuduclusterkey@0"
I20260927 06:56:27.035948 11678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:56:27.067788 11678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:56:27.071739 11678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:56:27.072057 11678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:56:27.090637 11678 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45147
I20260927 06:56:27.090673 11744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45147 every 8 connection(s)
I20260927 06:56:27.091940 11678 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:56:27.095577 11745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:56:27.098093 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 11678
I20260927 06:56:27.098296 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:56:27.098909 388 external_mini_cluster.cc:1428] Setting key 49ba38c910aefad4e36974188e44665d
I20260927 06:56:27.107683 11745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8: Bootstrap starting.
I20260927 06:56:27.111331 11745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8: Neither blocks nor log segments found. Creating new log.
I20260927 06:56:27.112632 11745 log.cc:824] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8: Log is configured to *not* fsync() on all Append() calls
Sep 27 06:56:27 dist-test-slave-f892 krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:56:27.115839 11745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8: No bootstrap required, opened a new log
I20260927 06:56:27.120854 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:27.101134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46364 (local address 127.0.97.62:45147)
0927 06:56:27.101870 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0927 06:56:27.101881 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:56:27.101926 (+ 45us) server_negotiation.cc:407] Connection header received
0927 06:56:27.103550 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:56:27.103570 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:56:27.103906 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:56:27.104224 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:56:27.105763 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:27.107067 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:27.108748 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:27.109389 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:27.109511 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:27.114097 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:56:27.114105 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:56:27.114123 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:56:27.114160 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:56:27.117107 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:56:27.117777 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:56:27.117795 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:56:27.117804 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:56:27.117967 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:56:27.118367 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:56:27.118374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:56:27.118377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:56:27.118852 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:56:27.119144 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 06:56:27.119596 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 06:56:27.119932 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":368,"threads_started":1}
I20260927 06:56:27.123857 11745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c92fdaf6a38425db931ec95d99b45c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45147 } }
I20260927 06:56:27.124214 11745 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:56:27.124312 11745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c92fdaf6a38425db931ec95d99b45c8, State: Initialized, Role: FOLLOWER
I20260927 06:56:27.124840 11745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c92fdaf6a38425db931ec95d99b45c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45147 } }
I20260927 06:56:27.125085 11745 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:56:27.125223 11745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:56:27.125387 11745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:56:27.128312 11745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c92fdaf6a38425db931ec95d99b45c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45147 } }
I20260927 06:56:27.128883 11745 leader_election.cc:302] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c92fdaf6a38425db931ec95d99b45c8; no voters:
I20260927 06:56:27.129465 11745 leader_election.cc:288] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:56:27.129560 11750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:56:27.130365 11750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8 [term 1 LEADER]: Becoming Leader. State: Replica: 5c92fdaf6a38425db931ec95d99b45c8, State: Running, Role: LEADER
I20260927 06:56:27.130878 11750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c92fdaf6a38425db931ec95d99b45c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45147 } }
I20260927 06:56:27.131516 11745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:56:27.133396 11752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c92fdaf6a38425db931ec95d99b45c8. Latest consensus state: current_term: 1 leader_uuid: "5c92fdaf6a38425db931ec95d99b45c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c92fdaf6a38425db931ec95d99b45c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45147 } } }
I20260927 06:56:27.133742 11752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8 [sys.catalog]: This master's current role is: LEADER
I20260927 06:56:27.136724 11755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:56:27.136610 11751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c92fdaf6a38425db931ec95d99b45c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c92fdaf6a38425db931ec95d99b45c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45147 } } }
I20260927 06:56:27.136976 11751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8 [sys.catalog]: This master's current role is: LEADER
I20260927 06:56:27.141947 11755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:56:27.142601 11745 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 06:56:27.159413 11755 catalog_manager.cc:1347] Generated new cluster ID: 7f5f35d0a14f4dca864e4fedd96646c5
I20260927 06:56:27.159611 11755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:56:27.173561 11755 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:56:27.175069 11755 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:56:27.182484 11755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8: Generated new TSK 0
I20260927 06:56:27.183275 11755 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 06:56:27.915580 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51399/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45147
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 06:56:28 dist-test-slave-f892 krb5kdc[11234](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 06:56:28.306505 11790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:56:28.306864 11790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:56:28.307006 11790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:56:28.318704 11790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:56:28.318861 11790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:56:28.319047 11790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:56:28.334847 11790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:51399/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:56:28.336364 11790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:56:28.338171 11790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:56:28.357062 11801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:56:28.359053 11800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:56:28.362534 11790 server_base.cc:1034] running on GCE node
W20260927 06:56:28.363253 11803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:56:28.364037 11790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:56:28.365195 11790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:56:28.366489 11790 hybrid_clock.cc:648] HybridClock initialized: now 1790492188366416 us; error 53 us; skew 500 ppm
I20260927 06:56:28.366881 11790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:56:28 dist-test-slave-f892 krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:56:28.371007 11790 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:56:28.372931 11790 webserver.cc:460] Webserver started at http://127.0.97.1:41487/ using document root <none> and password file <none>
I20260927 06:56:28.373857 11790 fs_manager.cc:359] Metadata directory not provided
I20260927 06:56:28.374030 11790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:56:28.374431 11790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 06:56:28 dist-test-slave-f892 krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 06:56:28.393956 11790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "a5357c9fd392461b8a04582756d78f07"
format_stamp: "Formatted at 2026-09-27 06:56:28 on dist-test-slave-f892"
server_key: "241d7f0d7fbc30b03ba523fd0195cacc"
server_key_iv: "0da6ffbb9e5f32783c843f7bf55afd32"
server_key_version: "kuduclusterkey@0"
I20260927 06:56:28.395174 11790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a5357c9fd392461b8a04582756d78f07"
format_stamp: "Formatted at 2026-09-27 06:56:28 on dist-test-slave-f892"
server_key: "241d7f0d7fbc30b03ba523fd0195cacc"
server_key_iv: "0da6ffbb9e5f32783c843f7bf55afd32"
server_key_version: "kuduclusterkey@0"
I20260927 06:56:28.403574 11790 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260927 06:56:28.431450 11810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:56:28.433481 11790 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.003s
I20260927 06:56:28.433766 11790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "a5357c9fd392461b8a04582756d78f07"
format_stamp: "Formatted at 2026-09-27 06:56:28 on dist-test-slave-f892"
server_key: "241d7f0d7fbc30b03ba523fd0195cacc"
server_key_iv: "0da6ffbb9e5f32783c843f7bf55afd32"
server_key_version: "kuduclusterkey@0"
I20260927 06:56:28.434160 11790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:56:28.488258 11790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:56:28.493240 11790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:56:28.493568 11790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:56:28.495218 11790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:56:28.496681 11790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:56:28.496780 11790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:56:28.496932 11790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:56:28.497017 11790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:56:28.640996 11922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42187 every 8 connection(s)
I20260927 06:56:28.641407 11790 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42187
I20260927 06:56:28.643311 11790 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:56:28.646270 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 11790
I20260927 06:56:28.646462 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:56:28.647279 388 external_mini_cluster.cc:1428] Setting key 0e37552755961a9a118f09d72bbfe0e6
Sep 27 06:56:28 dist-test-slave-f892 krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 06:56:28.718621 11927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:28.660782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59945 (local address 127.0.97.62:45147)
0927 06:56:28.685671 (+ 24889us) server_negotiation.cc:206] Beginning negotiation
0927 06:56:28.685677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:56:28.689452 (+ 3775us) server_negotiation.cc:407] Connection header received
0927 06:56:28.689520 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:56:28.689524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:56:28.689612 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:56:28.689709 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:56:28.698534 (+ 8825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:28.699667 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:28.704038 (+ 4371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:28.705117 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:28.705215 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:28.711543 (+ 6328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:56:28.711549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:56:28.711554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:56:28.711593 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:56:28.715085 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:56:28.715968 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:56:28.715974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:56:28.715978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:56:28.716123 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:56:28.716603 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:56:28.716609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:56:28.716615 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:56:28.716944 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:56:28.717063 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 06:56:28.717940 (+ 877us) server_negotiation.cc:299] Negotiation successful
0927 06:56:28.718190 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22651,"thread_start_us":22764,"threads_started":1}
I20260927 06:56:28.720732 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:28.660150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45147 (local address 127.0.97.1:59945)
0927 06:56:28.686352 (+ 26202us) negotiation.cc:107] Waiting for socket to connect
0927 06:56:28.686411 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 06:56:28.689010 (+ 2599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:56:28.694347 (+ 5337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:56:28.694359 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:56:28.694983 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:56:28.695754 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:56:28.695777 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:56:28.699891 (+ 4114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:56:28.699896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:56:28.701819 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:56:28.701835 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:56:28.705554 (+ 3719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:56:28.705561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:56:28.705866 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:28.706670 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:56:28.706700 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:56:28.711236 (+ 4536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:56:28.715256 (+ 4020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:56:28.715265 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:56:28.715280 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:56:28.715831 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:56:28.716259 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:56:28.716266 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:56:28.716272 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:56:28.716495 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:56:28.717145 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:56:28.717155 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:56:28.717556 (+ 401us) client_negotiation.cc:769] Sending connection context
0927 06:56:28.718899 (+ 1343us) client_negotiation.cc:240] Negotiation successful
0927 06:56:28.719419 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25845,"thread_start_us":31740,"threads_started":1}
I20260927 06:56:28.722538 11923 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45147
I20260927 06:56:28.722862 11923 heartbeater.cc:464] Registering TS with master...
I20260927 06:56:28.723647 11923 heartbeater.cc:511] Master 127.0.97.62:45147 requested a full tablet report, sending...
I20260927 06:56:28.726140 11710 ts_manager.cc:194] Registered new tserver with Master: a5357c9fd392461b8a04582756d78f07 (127.0.97.1:42187)
I20260927 06:56:28.728135 11710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260927 06:56:28.773597 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.123s user 0.004s sys 0.012s
I20260927 06:56:28.774591 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51399/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45147
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 06:56:28 dist-test-slave-f892 krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 06:56:29 dist-test-slave-f892 krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 06:56:29.300105 11929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:56:29.300573 11929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:56:29.300756 11929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:56:29.311733 11929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:56:29.311995 11929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:56:29.312229 11929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:56:29.327734 11929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:51399/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:56:29.329545 11929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:56:29.331267 11929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:56:29.359716 11936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:56:29.366652 11937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:56:29.376986 11939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:56:29.378613 11929 server_base.cc:1034] running on GCE node
I20260927 06:56:29.379581 11929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:56:29.383440 11929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:56:29.386866 11929 hybrid_clock.cc:648] HybridClock initialized: now 1790492189386752 us; error 102 us; skew 500 ppm
I20260927 06:56:29.387341 11929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:56:29 dist-test-slave-f892 krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:56:29.391924 11929 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:56:29.408057 11929 webserver.cc:460] Webserver started at http://127.0.97.2:45493/ using document root <none> and password file <none>
I20260927 06:56:29.409226 11929 fs_manager.cc:359] Metadata directory not provided
I20260927 06:56:29.409440 11929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:56:29.409869 11929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 06:56:29 dist-test-slave-f892 krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 06:56:29.441865 11929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "a2a7b77bddfa4b1fabb256862acb0834"
format_stamp: "Formatted at 2026-09-27 06:56:29 on dist-test-slave-f892"
server_key: "6cba4dc8d9be19557a4a9aee833a0be7"
server_key_iv: "6db4a38de72348befbf3baadb181e75f"
server_key_version: "kuduclusterkey@0"
I20260927 06:56:29.443532 11929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a2a7b77bddfa4b1fabb256862acb0834"
format_stamp: "Formatted at 2026-09-27 06:56:29 on dist-test-slave-f892"
server_key: "6cba4dc8d9be19557a4a9aee833a0be7"
server_key_iv: "6db4a38de72348befbf3baadb181e75f"
server_key_version: "kuduclusterkey@0"
I20260927 06:56:29.449765 11929 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 06:56:29.488772 11948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:56:29.490897 11929 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.003s sys 0.001s
I20260927 06:56:29.491170 11929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "a2a7b77bddfa4b1fabb256862acb0834"
format_stamp: "Formatted at 2026-09-27 06:56:29 on dist-test-slave-f892"
server_key: "6cba4dc8d9be19557a4a9aee833a0be7"
server_key_iv: "6db4a38de72348befbf3baadb181e75f"
server_key_version: "kuduclusterkey@0"
I20260927 06:56:29.491523 11929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:56:29.508047 11929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:56:29.512087 11929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:56:29.512392 11929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:56:29.513375 11929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:56:29.514925 11929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:56:29.515023 11929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:56:29.515177 11929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:56:29.515246 11929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:56:29.557983 11929 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34885
I20260927 06:56:29.558020 12065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34885 every 8 connection(s)
I20260927 06:56:29.559305 11929 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:56:29.561398 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 11929
I20260927 06:56:29.561563 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:56:29.562175 388 external_mini_cluster.cc:1428] Setting key 469067e2f394337f5060b0c4a91021cd
Sep 27 06:56:29 dist-test-slave-f892 krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 06:56:29.598815 12068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:29.562304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51597 (local address 127.0.97.62:45147)
0927 06:56:29.577737 (+ 15433us) server_negotiation.cc:206] Beginning negotiation
0927 06:56:29.577745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:56:29.577771 (+ 26us) server_negotiation.cc:407] Connection header received
0927 06:56:29.579311 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:56:29.579316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:56:29.579429 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:56:29.579538 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:56:29.581689 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:29.582836 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:29.585987 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:29.586627 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:29.586770 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:29.591308 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:56:29.591318 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:56:29.591326 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:56:29.591384 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:56:29.595243 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:56:29.596216 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:56:29.596225 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:56:29.596231 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:56:29.596407 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:56:29.596918 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:56:29.596923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:56:29.596927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:56:29.597415 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:56:29.597528 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 06:56:29.598400 (+ 872us) server_negotiation.cc:299] Negotiation successful
0927 06:56:29.598564 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15293,"thread_start_us":15473,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 06:56:29.600131 12069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:29.562652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45147 (local address 127.0.97.2:51597)
0927 06:56:29.576307 (+ 13655us) negotiation.cc:107] Waiting for socket to connect
0927 06:56:29.576436 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0927 06:56:29.578942 (+ 2506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:56:29.580339 (+ 1397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:56:29.580349 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:56:29.580805 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:56:29.581456 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:56:29.581486 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:56:29.583035 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:56:29.583043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:56:29.585788 (+ 2745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:56:29.585808 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:56:29.586869 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:56:29.586878 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:56:29.587202 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:29.588000 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:56:29.588032 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:56:29.591052 (+ 3020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:56:29.595395 (+ 4343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:56:29.595405 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:56:29.595422 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:56:29.596014 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:56:29.596541 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:56:29.596546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:56:29.596550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:56:29.596774 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:56:29.597567 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:56:29.597575 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:56:29.597980 (+ 405us) client_negotiation.cc:769] Sending connection context
0927 06:56:29.598316 (+ 336us) client_negotiation.cc:240] Negotiation successful
0927 06:56:29.598812 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11872,"thread_start_us":16587,"threads_started":1}
I20260927 06:56:29.601605 12066 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45147
I20260927 06:56:29.601922 12066 heartbeater.cc:464] Registering TS with master...
I20260927 06:56:29.602790 12066 heartbeater.cc:511] Master 127.0.97.62:45147 requested a full tablet report, sending...
I20260927 06:56:29.604485 11710 ts_manager.cc:194] Registered new tserver with Master: a2a7b77bddfa4b1fabb256862acb0834 (127.0.97.2:34885)
I20260927 06:56:29.605562 11710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 06:56:29.627395 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51399/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45147
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:56:29.731949 11923 heartbeater.cc:503] Master 127.0.97.62:45147 was elected leader, sending a full tablet report...
W20260927 06:56:29.988487 12072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:56:29.988837 12072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:56:29.988997 12072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:56:29.996949 12072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:56:29.997123 12072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:56:29.997284 12072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:56:30.007138 12072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:51399/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:56:30.008333 12072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:56:30.009603 12072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:56:30.023095 12078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:56:30.023175 12080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:56:30.023152 12077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:56:30.023844 12072 server_base.cc:1034] running on GCE node
I20260927 06:56:30.024477 12072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:56:30.025404 12072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:56:30.026623 12072 hybrid_clock.cc:648] HybridClock initialized: now 1790492190026577 us; error 57 us; skew 500 ppm
I20260927 06:56:30.026937 12072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:56:30 dist-test-slave-f892 krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:56:30.029975 12072 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:56:30.031939 12072 webserver.cc:460] Webserver started at http://127.0.97.3:39011/ using document root <none> and password file <none>
I20260927 06:56:30.032600 12072 fs_manager.cc:359] Metadata directory not provided
I20260927 06:56:30.032737 12072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:56:30.033068 12072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 06:56:30 dist-test-slave-f892 krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 06:56:30.048969 12072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "4397721a147849d8ab6e5ac3a1922157"
format_stamp: "Formatted at 2026-09-27 06:56:30 on dist-test-slave-f892"
server_key: "71f5a9460fb21750b40db405c08de981"
server_key_iv: "7c19afa00bb4f1353d6b09a16daca232"
server_key_version: "kuduclusterkey@0"
I20260927 06:56:30.050114 12072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4397721a147849d8ab6e5ac3a1922157"
format_stamp: "Formatted at 2026-09-27 06:56:30 on dist-test-slave-f892"
server_key: "71f5a9460fb21750b40db405c08de981"
server_key_iv: "7c19afa00bb4f1353d6b09a16daca232"
server_key_version: "kuduclusterkey@0"
I20260927 06:56:30.056496 12072 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 06:56:30.074489 12087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:56:30.076282 12072 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 06:56:30.076431 12072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "4397721a147849d8ab6e5ac3a1922157"
format_stamp: "Formatted at 2026-09-27 06:56:30 on dist-test-slave-f892"
server_key: "71f5a9460fb21750b40db405c08de981"
server_key_iv: "7c19afa00bb4f1353d6b09a16daca232"
server_key_version: "kuduclusterkey@0"
I20260927 06:56:30.076622 12072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:56:30.107632 12072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:56:30.111740 12072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:56:30.112025 12072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:56:30.112879 12072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:56:30.114192 12072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:56:30.114262 12072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:56:30.114341 12072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:56:30.114384 12072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:56:30.147737 12072 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46381
I20260927 06:56:30.147753 12199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46381 every 8 connection(s)
I20260927 06:56:30.148938 12072 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:56:30.150614 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 12072
I20260927 06:56:30.150789 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:56:30.151407 388 external_mini_cluster.cc:1428] Setting key 5bdf836c25983d7a9e279e2feaa7c3ab
Sep 27 06:56:30 dist-test-slave-f892 krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:56:30.173988 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.151687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33869 (local address 127.0.97.62:45147)
0927 06:56:30.152589 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0927 06:56:30.152595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:56:30.153397 (+ 802us) server_negotiation.cc:407] Connection header received
0927 06:56:30.155113 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:56:30.155117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:56:30.155174 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:56:30.155259 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:56:30.156541 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.157500 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.159742 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.160931 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.161070 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:30.165460 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:56:30.165466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:56:30.165472 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:56:30.165549 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:56:30.170393 (+ 4844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:56:30.171475 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:56:30.171482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:56:30.171487 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:56:30.171638 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:56:30.172250 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:56:30.172255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:56:30.172258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:56:30.172603 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:56:30.172753 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 06:56:30.173423 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 06:56:30.173634 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":636,"threads_started":1}
I20260927 06:56:30.175024 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.152234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45147 (local address 127.0.97.3:33869)
0927 06:56:30.153271 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0927 06:56:30.153314 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 06:56:30.154885 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:56:30.155464 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:56:30.155472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:56:30.155883 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:56:30.156399 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:56:30.156415 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.157711 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.157718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:56:30.159584 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:56:30.159598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.161162 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.161172 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:56:30.161427 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:30.162142 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:56:30.162164 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:56:30.165066 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:56:30.170560 (+ 5494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:56:30.170571 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:56:30.170591 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:56:30.171321 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:56:30.171752 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:56:30.171756 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:56:30.171760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:56:30.172115 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:56:30.172727 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:56:30.172735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:56:30.173072 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 06:56:30.173326 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 06:56:30.173821 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":573,"threads_started":1}
I20260927 06:56:30.176750 12200 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45147
I20260927 06:56:30.177125 12200 heartbeater.cc:464] Registering TS with master...
I20260927 06:56:30.179538 12200 heartbeater.cc:511] Master 127.0.97.62:45147 requested a full tablet report, sending...
I20260927 06:56:30.184891 11709 ts_manager.cc:194] Registered new tserver with Master: 4397721a147849d8ab6e5ac3a1922157 (127.0.97.3:46381)
I20260927 06:56:30.186226 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:56:30.186197 11709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33869
I20260927 06:56:30.208333 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.191327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.0.97.62:45147)
0927 06:56:30.191490 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 06:56:30.191496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:56:30.192251 (+ 755us) server_negotiation.cc:407] Connection header received
0927 06:56:30.192483 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:56:30.192490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:56:30.192597 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:56:30.192703 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:56:30.194171 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.195047 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.197218 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.197849 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.197951 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:30.200997 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:56:30.201003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:56:30.201008 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:56:30.201044 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:56:30.204746 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:56:30.205689 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:56:30.205694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:56:30.205698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:56:30.205902 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:56:30.206401 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:56:30.206408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:56:30.206427 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:56:30.206834 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:56:30.207115 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 06:56:30.207952 (+ 837us) server_negotiation.cc:299] Negotiation successful
0927 06:56:30.208109 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 06:56:30.212756 11709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:56:30.220157 11709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:56:30.244587 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.231751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46381 (local address 127.0.0.1:38470)
0927 06:56:30.232934 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0927 06:56:30.232977 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 06:56:30.233145 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:56:30.233965 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:56:30.233979 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:56:30.234045 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:56:30.235093 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:56:30.235120 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.237822 (+ 2702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.237831 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:56:30.240952 (+ 3121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:56:30.240970 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.243471 (+ 2501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.243480 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:56:30.243746 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:30.243826 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 06:56:30.243941 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 06:56:30.244174 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"threads_started":1}
I20260927 06:56:30.245532 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.232198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38470 (local address 127.0.97.3:46381)
0927 06:56:30.232946 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0927 06:56:30.232958 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:56:30.233055 (+ 97us) server_negotiation.cc:407] Connection header received
0927 06:56:30.233378 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:56:30.233389 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:56:30.233669 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:56:30.235361 (+ 1692us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:56:30.235660 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.237629 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.241224 (+ 3595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.243195 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.244770 (+ 1575us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:30.244982 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 06:56:30.245048 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 06:56:30.245194 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":415,"threads_started":1}
I20260927 06:56:30.247946 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.232268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42187 (local address 127.0.0.1:36134)
0927 06:56:30.234209 (+ 1941us) negotiation.cc:107] Waiting for socket to connect
0927 06:56:30.234227 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 06:56:30.234425 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:56:30.236066 (+ 1641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:56:30.236073 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:56:30.236108 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:56:30.236558 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:56:30.236571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.240410 (+ 3839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.240418 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:56:30.244806 (+ 4388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:56:30.244835 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.247178 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.247187 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:56:30.247363 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:30.247393 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 06:56:30.247493 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 06:56:30.247625 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1803,"thread_start_us":675,"threads_started":1}
I20260927 06:56:30.247993 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.232826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36134 (local address 127.0.97.1:42187)
0927 06:56:30.233706 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0927 06:56:30.233718 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:56:30.234327 (+ 609us) server_negotiation.cc:407] Connection header received
0927 06:56:30.234618 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:56:30.234625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:56:30.234846 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:56:30.234997 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:56:30.236765 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.240189 (+ 3424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.245038 (+ 4849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.246695 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.246815 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:30.247028 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 06:56:30.247557 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 06:56:30.247701 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":1670,"threads_started":1}
I20260927 06:56:30.249212 12135 tablet_service.cc:1467] Processing CreateTablet for tablet 199cfab446dc403dbb9dc3185d6e398d (DEFAULT_TABLE table=test-table [id=73f8840034c145a49ad242ae5e08c2e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:56:30.250456 12135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 199cfab446dc403dbb9dc3185d6e398d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:56:30.252013 11858 tablet_service.cc:1467] Processing CreateTablet for tablet 199cfab446dc403dbb9dc3185d6e398d (DEFAULT_TABLE table=test-table [id=73f8840034c145a49ad242ae5e08c2e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:56:30.253461 11858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 199cfab446dc403dbb9dc3185d6e398d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:56:30.253893 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.233652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34885 (local address 127.0.0.1:33492)
0927 06:56:30.244665 (+ 11013us) negotiation.cc:107] Waiting for socket to connect
0927 06:56:30.244681 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:56:30.245340 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:56:30.246644 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:56:30.246648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:56:30.246671 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:56:30.246946 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:56:30.246957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.248741 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.248747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:56:30.251131 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:56:30.251146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.253177 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.253182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:56:30.253344 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:30.253374 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 06:56:30.253463 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 06:56:30.253586 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10874,"thread_start_us":996,"threads_started":1}
I20260927 06:56:30.254158 12218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.234321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33492 (local address 127.0.97.2:34885)
0927 06:56:30.245888 (+ 11567us) server_negotiation.cc:206] Beginning negotiation
0927 06:56:30.245895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:56:30.245920 (+ 25us) server_negotiation.cc:407] Connection header received
0927 06:56:30.246031 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:56:30.246039 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:56:30.246243 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:56:30.246415 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:56:30.247125 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.248611 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.251314 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.252713 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.252846 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:30.253036 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 06:56:30.253672 (+ 636us) server_negotiation.cc:299] Negotiation successful
0927 06:56:30.253835 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11412,"thread_start_us":11194,"threads_started":1}
I20260927 06:56:30.257417 12001 tablet_service.cc:1467] Processing CreateTablet for tablet 199cfab446dc403dbb9dc3185d6e398d (DEFAULT_TABLE table=test-table [id=73f8840034c145a49ad242ae5e08c2e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:56:30.258752 12001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 199cfab446dc403dbb9dc3185d6e398d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:56:30.262104 12219 tablet_bootstrap.cc:492] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157: Bootstrap starting.
I20260927 06:56:30.265319 12219 tablet_bootstrap.cc:654] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157: Neither blocks nor log segments found. Creating new log.
I20260927 06:56:30.266597 12219 log.cc:824] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157: Log is configured to *not* fsync() on all Append() calls
I20260927 06:56:30.270151 12220 tablet_bootstrap.cc:492] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07: Bootstrap starting.
I20260927 06:56:30.271003 12219 tablet_bootstrap.cc:492] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157: No bootstrap required, opened a new log
I20260927 06:56:30.271265 12219 ts_tablet_manager.cc:1397] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260927 06:56:30.273726 12220 tablet_bootstrap.cc:654] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07: Neither blocks nor log segments found. Creating new log.
I20260927 06:56:30.274920 12220 log.cc:824] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07: Log is configured to *not* fsync() on all Append() calls
I20260927 06:56:30.278492 12220 tablet_bootstrap.cc:492] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07: No bootstrap required, opened a new log
I20260927 06:56:30.278745 12220 ts_tablet_manager.cc:1397] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 06:56:30.278764 12222 tablet_bootstrap.cc:492] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834: Bootstrap starting.
I20260927 06:56:30.281531 12219 raft_consensus.cc:348] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397721a147849d8ab6e5ac3a1922157" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46381 } } peers { permanent_uuid: "a5357c9fd392461b8a04582756d78f07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42187 } } peers { permanent_uuid: "a2a7b77bddfa4b1fabb256862acb0834" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34885 } }
I20260927 06:56:30.281950 12219 raft_consensus.cc:374] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:56:30.281980 12222 tablet_bootstrap.cc:654] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834: Neither blocks nor log segments found. Creating new log.
I20260927 06:56:30.282064 12219 raft_consensus.cc:729] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397721a147849d8ab6e5ac3a1922157, State: Initialized, Role: FOLLOWER
I20260927 06:56:30.282671 12219 consensus_queue.cc:260] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397721a147849d8ab6e5ac3a1922157" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46381 } } peers { permanent_uuid: "a5357c9fd392461b8a04582756d78f07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42187 } } peers { permanent_uuid: "a2a7b77bddfa4b1fabb256862acb0834" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34885 } }
I20260927 06:56:30.284188 12222 log.cc:824] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834: Log is configured to *not* fsync() on all Append() calls
I20260927 06:56:30.285480 12200 heartbeater.cc:503] Master 127.0.97.62:45147 was elected leader, sending a full tablet report...
I20260927 06:56:30.286159 12219 ts_tablet_manager.cc:1428] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260927 06:56:30.287905 12222 tablet_bootstrap.cc:492] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834: No bootstrap required, opened a new log
I20260927 06:56:30.288183 12222 ts_tablet_manager.cc:1397] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 06:56:30.289599 12220 raft_consensus.cc:348] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397721a147849d8ab6e5ac3a1922157" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46381 } } peers { permanent_uuid: "a5357c9fd392461b8a04582756d78f07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42187 } } peers { permanent_uuid: "a2a7b77bddfa4b1fabb256862acb0834" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34885 } }
I20260927 06:56:30.289960 12220 raft_consensus.cc:374] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:56:30.290119 12220 raft_consensus.cc:729] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5357c9fd392461b8a04582756d78f07, State: Initialized, Role: FOLLOWER
I20260927 06:56:30.290735 12220 consensus_queue.cc:260] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397721a147849d8ab6e5ac3a1922157" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46381 } } peers { permanent_uuid: "a5357c9fd392461b8a04582756d78f07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42187 } } peers { permanent_uuid: "a2a7b77bddfa4b1fabb256862acb0834" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34885 } }
I20260927 06:56:30.292430 12220 ts_tablet_manager.cc:1428] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 06:56:30.297631 12222 raft_consensus.cc:348] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397721a147849d8ab6e5ac3a1922157" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46381 } } peers { permanent_uuid: "a5357c9fd392461b8a04582756d78f07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42187 } } peers { permanent_uuid: "a2a7b77bddfa4b1fabb256862acb0834" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34885 } }
I20260927 06:56:30.297950 12222 raft_consensus.cc:374] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:56:30.298087 12222 raft_consensus.cc:729] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a7b77bddfa4b1fabb256862acb0834, State: Initialized, Role: FOLLOWER
I20260927 06:56:30.298626 12222 consensus_queue.cc:260] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397721a147849d8ab6e5ac3a1922157" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46381 } } peers { permanent_uuid: "a5357c9fd392461b8a04582756d78f07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42187 } } peers { permanent_uuid: "a2a7b77bddfa4b1fabb256862acb0834" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34885 } }
I20260927 06:56:30.299484 12066 heartbeater.cc:503] Master 127.0.97.62:45147 was elected leader, sending a full tablet report...
I20260927 06:56:30.300071 12222 ts_tablet_manager.cc:1428] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260927 06:56:30.311228 12067 tablet.cc:2369] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:56:30.401664 12201 tablet.cc:2369] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:56:30.404536 12226 raft_consensus.cc:482] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:56:30.404826 12226 raft_consensus.cc:504] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397721a147849d8ab6e5ac3a1922157" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46381 } } peers { permanent_uuid: "a5357c9fd392461b8a04582756d78f07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42187 } } peers { permanent_uuid: "a2a7b77bddfa4b1fabb256862acb0834" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34885 } }
I20260927 06:56:30.406524 12226 leader_election.cc:288] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4397721a147849d8ab6e5ac3a1922157 (127.0.97.3:46381), a2a7b77bddfa4b1fabb256862acb0834 (127.0.97.2:34885)
I20260927 06:56:30.413571 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.406922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59285 (local address 127.0.97.3:46381)
0927 06:56:30.407138 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 06:56:30.407147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:56:30.407861 (+ 714us) server_negotiation.cc:407] Connection header received
0927 06:56:30.408019 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:56:30.408025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:56:30.408097 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:56:30.408188 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:56:30.408870 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.409666 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.411703 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.412737 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.412828 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:30.412900 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 06:56:30.413273 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 06:56:30.413351 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 06:56:30.413746 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.406870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46381 (local address 127.0.97.1:59285)
0927 06:56:30.407737 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0927 06:56:30.407757 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 06:56:30.407877 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:56:30.408233 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:56:30.408238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:56:30.408266 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:56:30.408703 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:56:30.408714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.409791 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.409796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:56:30.411553 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:56:30.411564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.412929 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.412935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:56:30.413112 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:30.413148 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 06:56:30.413249 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 06:56:30.413386 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":550,"threads_started":1}
I20260927 06:56:30.414580 12218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.406992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40705 (local address 127.0.97.2:34885)
0927 06:56:30.407206 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 06:56:30.407215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:56:30.408064 (+ 849us) server_negotiation.cc:407] Connection header received
0927 06:56:30.408244 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:56:30.408250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:56:30.408341 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:56:30.408454 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:56:30.409117 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.410547 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.412414 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.413398 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.413522 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:30.413611 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 06:56:30.414219 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 06:56:30.414339 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 06:56:30.414681 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.406872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34885 (local address 127.0.97.1:40705)
0927 06:56:30.407908 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0927 06:56:30.407945 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 06:56:30.408074 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:56:30.408519 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:56:30.408522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:56:30.408543 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:56:30.408928 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:56:30.408941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.410675 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.410679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:56:30.412258 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:56:30.412272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.413619 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.413715 (+ 96us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:56:30.414046 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:30.414086 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 06:56:30.414182 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 06:56:30.414296 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":53,"thread_start_us":454,"threads_started":1}
I20260927 06:56:30.414768 12155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199cfab446dc403dbb9dc3185d6e398d" candidate_uuid: "a5357c9fd392461b8a04582756d78f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397721a147849d8ab6e5ac3a1922157" is_pre_election: true
I20260927 06:56:30.415282 12155 raft_consensus.cc:2454] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5357c9fd392461b8a04582756d78f07 in term 0.
I20260927 06:56:30.415477 12021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199cfab446dc403dbb9dc3185d6e398d" candidate_uuid: "a5357c9fd392461b8a04582756d78f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a7b77bddfa4b1fabb256862acb0834" is_pre_election: true
I20260927 06:56:30.415962 12021 raft_consensus.cc:2454] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5357c9fd392461b8a04582756d78f07 in term 0.
I20260927 06:56:30.415925 11813 leader_election.cc:302] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397721a147849d8ab6e5ac3a1922157, a5357c9fd392461b8a04582756d78f07; no voters:
I20260927 06:56:30.416535 12226 raft_consensus.cc:2781] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:56:30.416721 12226 raft_consensus.cc:482] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:56:30.416854 12226 raft_consensus.cc:3037] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:56:30.419118 12226 raft_consensus.cc:504] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397721a147849d8ab6e5ac3a1922157" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46381 } } peers { permanent_uuid: "a5357c9fd392461b8a04582756d78f07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42187 } } peers { permanent_uuid: "a2a7b77bddfa4b1fabb256862acb0834" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34885 } }
I20260927 06:56:30.419731 12226 leader_election.cc:288] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [CANDIDATE]: Term 1 election: Requested vote from peers 4397721a147849d8ab6e5ac3a1922157 (127.0.97.3:46381), a2a7b77bddfa4b1fabb256862acb0834 (127.0.97.2:34885)
I20260927 06:56:30.420156 12021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199cfab446dc403dbb9dc3185d6e398d" candidate_uuid: "a5357c9fd392461b8a04582756d78f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a7b77bddfa4b1fabb256862acb0834"
I20260927 06:56:30.420189 12155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199cfab446dc403dbb9dc3185d6e398d" candidate_uuid: "a5357c9fd392461b8a04582756d78f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397721a147849d8ab6e5ac3a1922157"
I20260927 06:56:30.420334 12021 raft_consensus.cc:3037] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:56:30.420435 12155 raft_consensus.cc:3037] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:56:30.422914 12155 raft_consensus.cc:2454] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5357c9fd392461b8a04582756d78f07 in term 1.
I20260927 06:56:30.423352 12021 raft_consensus.cc:2454] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5357c9fd392461b8a04582756d78f07 in term 1.
I20260927 06:56:30.423405 11813 leader_election.cc:302] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397721a147849d8ab6e5ac3a1922157, a5357c9fd392461b8a04582756d78f07; no voters:
I20260927 06:56:30.423703 12226 raft_consensus.cc:2781] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:56:30.424203 12226 raft_consensus.cc:686] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [term 1 LEADER]: Becoming Leader. State: Replica: a5357c9fd392461b8a04582756d78f07, State: Running, Role: LEADER
I20260927 06:56:30.424728 12226 consensus_queue.cc:237] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397721a147849d8ab6e5ac3a1922157" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46381 } } peers { permanent_uuid: "a5357c9fd392461b8a04582756d78f07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42187 } } peers { permanent_uuid: "a2a7b77bddfa4b1fabb256862acb0834" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34885 } }
I20260927 06:56:30.428690 11709 catalog_manager.cc:5495] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5357c9fd392461b8a04582756d78f07 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a5357c9fd392461b8a04582756d78f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397721a147849d8ab6e5ac3a1922157" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5357c9fd392461b8a04582756d78f07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a7b77bddfa4b1fabb256862acb0834" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34885 } health_report { overall_health: UNKNOWN } } }
W20260927 06:56:30.444772 11924 tablet.cc:2369] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:56:30.480834 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.475352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36140 (local address 127.0.97.1:42187)
0927 06:56:30.475569 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 06:56:30.475576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:56:30.475596 (+ 20us) server_negotiation.cc:407] Connection header received
0927 06:56:30.475731 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:56:30.475734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:56:30.475825 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:56:30.476002 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:56:30.476363 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.477084 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.478652 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.479072 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.479176 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:30.479649 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:56:30.480256 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:56:30.480391 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 06:56:30.480551 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 06:56:30.480614 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 06:56:30.489667 12021 raft_consensus.cc:1261] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834 [term 1 FOLLOWER]: Refusing update from remote peer a5357c9fd392461b8a04582756d78f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:56:30.489657 12155 raft_consensus.cc:1261] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157 [term 1 FOLLOWER]: Refusing update from remote peer a5357c9fd392461b8a04582756d78f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:56:30.490880 12226 consensus_queue.cc:1035] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4397721a147849d8ab6e5ac3a1922157" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:56:30.491247 12230 consensus_queue.cc:1035] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a7b77bddfa4b1fabb256862acb0834" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:56:30.505016 12233 mvcc.cc:205] Tried to move back new op lower bound from 7333856012234088448 to 7333856011982692352. Current Snapshot: MvccSnapshot[applied={T|T < 7333856012234088448}]
I20260927 06:56:30.514228 12234 mvcc.cc:205] Tried to move back new op lower bound from 7333856012234088448 to 7333856011982692352. Current Snapshot: MvccSnapshot[applied={T|T < 7333856012234088448 or (T in {7333856012234088448})}]
I20260927 06:56:30.519481 11710 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:56:30.519754 11710 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:56:30.523319 11710 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c92fdaf6a38425db931ec95d99b45c8: Sending DeleteTablet for 3 replicas of tablet 199cfab446dc403dbb9dc3185d6e398d
I20260927 06:56:30.524415 12135 tablet_service.cc:1514] Processing DeleteTablet for tablet 199cfab446dc403dbb9dc3185d6e398d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:56:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38470
I20260927 06:56:30.524611 11858 tablet_service.cc:1514] Processing DeleteTablet for tablet 199cfab446dc403dbb9dc3185d6e398d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:56:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36134
I20260927 06:56:30.524683 12001 tablet_service.cc:1514] Processing DeleteTablet for tablet 199cfab446dc403dbb9dc3185d6e398d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:56:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33492
I20260927 06:56:30.527063 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 11790
I20260927 06:56:30.536769 12243 tablet_replica.cc:330] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157: stopping tablet replica
I20260927 06:56:30.537431 12243 raft_consensus.cc:2229] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:56:30.538568 12243 raft_consensus.cc:2258] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:56:30.542640 12245 tablet_replica.cc:330] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834: stopping tablet replica
I20260927 06:56:30.542788 12243 ts_tablet_manager.cc:1905] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:56:30.543133 12245 raft_consensus.cc:2229] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:56:30.543691 12245 raft_consensus.cc:2258] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:56:30.549635 12243 ts_tablet_manager.cc:1918] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:56:30.549830 12243 log.cc:1196] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/199cfab446dc403dbb9dc3185d6e398d
I20260927 06:56:30.550429 12243 ts_tablet_manager.cc:1939] T 199cfab446dc403dbb9dc3185d6e398d P 4397721a147849d8ab6e5ac3a1922157: Deleting consensus metadata
I20260927 06:56:30.552233 12245 ts_tablet_manager.cc:1905] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:56:30.552737 11696 catalog_manager.cc:4841] TS 4397721a147849d8ab6e5ac3a1922157 (127.0.97.3:46381): tablet 199cfab446dc403dbb9dc3185d6e398d (table test-table [id=73f8840034c145a49ad242ae5e08c2e1]) successfully deleted
Sep 27 06:56:30 dist-test-slave-f892 krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:56:30.557925 12244 tablet_replica.cc:330] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07: stopping tablet replica
I20260927 06:56:30.558338 12244 raft_consensus.cc:2229] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:56:30.559142 12244 raft_consensus.cc:2258] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:56:30.562428 12245 ts_tablet_manager.cc:1918] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:56:30.562618 12245 log.cc:1196] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/199cfab446dc403dbb9dc3185d6e398d
I20260927 06:56:30.563196 12245 ts_tablet_manager.cc:1939] T 199cfab446dc403dbb9dc3185d6e398d P a2a7b77bddfa4b1fabb256862acb0834: Deleting consensus metadata
I20260927 06:56:30.565166 11697 catalog_manager.cc:4841] TS a2a7b77bddfa4b1fabb256862acb0834 (127.0.97.2:34885): tablet 199cfab446dc403dbb9dc3185d6e398d (table test-table [id=73f8840034c145a49ad242ae5e08c2e1]) successfully deleted
I20260927 06:56:30.568751 12244 ts_tablet_manager.cc:1905] T 199cfab446dc403dbb9dc3185d6e398d P a5357c9fd392461b8a04582756d78f07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:56:30.568815 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.528594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.0.97.1:42187)
0927 06:56:30.528765 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 06:56:30.528772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:56:30.539584 (+ 10812us) server_negotiation.cc:407] Connection header received
0927 06:56:30.540403 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:56:30.540413 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:56:30.540538 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:56:30.540618 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:56:30.545052 (+ 4434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.546657 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.549316 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.550330 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.550452 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:30.559271 (+ 8819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:56:30.559286 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:56:30.559302 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:56:30.559345 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:56:30.563468 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:56:30.564438 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:56:30.564448 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:56:30.564452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:56:30.564754 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:56:30.565455 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:56:30.565465 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:56:30.565469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:56:30.566023 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:56:30.566279 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 06:56:30.566826 (+ 547us) server_negotiation.cc:299] Negotiation successful
0927 06:56:30.567215 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 06:56:30.581017 11918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:56:30.689450 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 11790
I20260927 06:56:30.708320 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 11929
W20260927 06:56:30.708387 11697 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42187 (error 108)
I20260927 06:56:30.710201 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.709751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42187 (local address 127.0.0.1:36164)
0927 06:56:30.709906 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 06:56:30.710053 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 06:56:30.710458 11697 catalog_manager.cc:4584] TS a5357c9fd392461b8a04582756d78f07 (127.0.97.1:42187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 199cfab446dc403dbb9dc3185d6e398d: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42187: connect: Connection refused (error 111)
Sep 27 06:56:30 dist-test-slave-f892 krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:56:30.724421 12218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.709165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33502 (local address 127.0.97.2:34885)
0927 06:56:30.709350 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 06:56:30.709357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:56:30.709483 (+ 126us) server_negotiation.cc:407] Connection header received
0927 06:56:30.709659 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:56:30.709663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:56:30.709737 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:56:30.709867 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:56:30.711278 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.712044 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.713373 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.713748 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.713884 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:30.718856 (+ 4972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:56:30.718871 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:56:30.718887 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:56:30.718922 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:56:30.721634 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:56:30.722387 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:56:30.722396 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:56:30.722398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:56:30.722520 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:56:30.722897 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:56:30.722906 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:56:30.722908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:56:30.723252 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:56:30.723539 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 06:56:30.723846 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 06:56:30.724053 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 06:56:30.725037 12061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:56:30.764217 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.763698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42187 (local address 127.0.0.1:36180)
0927 06:56:30.763995 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 06:56:30.764085 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260927 06:56:30.765183 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.764850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42187 (local address 127.0.0.1:36184)
0927 06:56:30.765004 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 06:56:30.765067 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 06:56:30.817581 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 11929
I20260927 06:56:30.825634 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.825048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42187 (local address 127.0.0.1:36192)
0927 06:56:30.825287 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 06:56:30.825414 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 06:56:30.826848 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.826420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42187 (local address 127.0.0.1:36198)
0927 06:56:30.826600 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 06:56:30.826695 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 06:56:30.835125 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 12072
Sep 27 06:56:30 dist-test-slave-f892 krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 06:56:30.848912 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.835818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38478 (local address 127.0.97.3:46381)
0927 06:56:30.835973 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 06:56:30.835977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:56:30.835995 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:56:30.836129 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:56:30.836133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:56:30.836195 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:56:30.836309 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:56:30.837550 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.838356 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.839609 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:56:30.839980 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:56:30.840073 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:56:30.844108 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:56:30.844119 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:56:30.844135 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:56:30.844163 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:56:30.846699 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:56:30.847308 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:56:30.847314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:56:30.847317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:56:30.847429 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:56:30.847747 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:56:30.847754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:56:30.847758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:56:30.848092 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:56:30.848308 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 06:56:30.848532 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 06:56:30.848672 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 06:56:30.849519 12195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:56:30.907373 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.906935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42187 (local address 127.0.0.1:36214)
0927 06:56:30.907132 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 06:56:30.907212 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 06:56:30.908381 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:30.908059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42187 (local address 127.0.0.1:36226)
0927 06:56:30.908221 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 06:56:30.908293 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 06:56:30.948777 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 12072
I20260927 06:56:30.964736 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 11678
I20260927 06:56:30.968714 11740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:56:31.095376 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:56:31.094844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42187 (local address 127.0.0.1:36238)
0927 06:56:31.095046 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 06:56:31.095145 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 06:56:31.095963 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 11678
I20260927 06:56:31.119632 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 06:56:31.454608 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 06:56:31.454869 388 mini_ranger.cc:66] Stopping Ranger...
I20260927 06:56:31.800143 388 mini_ranger.cc:68] Stopped Ranger
2026-09-27 06:56:31.800 UTC [11264] LOG: received smart shutdown request
2026-09-27 06:56:31.803 UTC [11264] LOG: background worker "logical replication launcher" (PID 11273) exited with exit code 1
2026-09-27 06:56:31.804 UTC [11268] LOG: shutting down
2026-09-27 06:56:31.994 UTC [11264] LOG: database system is shut down
2026-09-27T06:56:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47907 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:56:32 dist-test-slave-f892 krb5kdc[12260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:56:32 dist-test-slave-f892 krb5kdc[12260](info): set up 2 sockets
Sep 27 06:56:32 dist-test-slave-f892 krb5kdc[12260](info): commencing operation
krb5kdc: starting...
W20260927 06:56:33.294338 388 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:56:33 dist-test-slave-f892 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:56:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:56:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260927 06:56:33.608676 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasktaM5HI/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/postgres -l logfile start
2026-09-27 06:56:36.352 UTC [12290] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 06:56:36.352 UTC [12290] LOG: listening on IPv4 address "127.0.97.20", port 55811
2026-09-27 06:56:36.356 UTC [12290] LOG: listening on Unix socket "/tmp/.s.PGSQL.55811"
2026-09-27 06:56:36.384 UTC [12293] LOG: database system was shut down at 2026-09-27 06:56:34 UTC
2026-09-27 06:56:36.388 UTC [12290] LOG: database system is ready to accept connections
I20260927 06:56:37.397974 388 mini_postgres.cc:96] Postgres bound to 55811
2026-09-27 06:56:37.414 UTC [12301] FATAL: database "slave" does not exist
127.0.97.20:55811 - accepting connections
I20260927 06:56:37.415052 388 mini_ranger.cc:161] Starting Ranger...
I20260927 06:56:37.439215 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 06:56:37.952755 388 mini_ranger.cc:87] Created ranger Postgres database
I20260927 06:56:37.952875 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-admin
2026-09-27 06:56:38,250 [I] DB FLAVOR :POSTGRES
2026-09-27 06:56:38,251 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 06:56:38,251 [I] Checking connection..
2026-09-27 06:56:38,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 06:56:38,522 [I] Checking connection passed.
2026-09-27 06:56:38,522 [I] --------- Verifying version history table ---------
2026-09-27 06:56:38,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 06:56:38,808 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 06:56:38,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 06:56:39,064 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 06:56:39,065 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 06:56:39,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 06:56:39.325 UTC [12365] WARNING: there is no transaction in progress
2026-09-27 06:56:39,331 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 06:56:39,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 06:56:39,596 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 06:56:39,597 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 06:56:39,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 06:56:39,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 06:56:40,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f892.c.gcp-upstream.internal','N') ;"
2026-09-27 06:56:40,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 06:56:40,624 [I] Table x_portal_user does not exist in database ranger
2026-09-27 06:56:40,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 06:56:40,887 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 06:56:40,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 06:56:41,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 06:56:41,391 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 06:56:41,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 06:56:42.427 UTC [12473] WARNING: there is no transaction in progress
2026-09-27 06:56:42.442 UTC [12473] WARNING: there is no transaction in progress
2026-09-27 06:56:42.456 UTC [12473] WARNING: there is no transaction in progress
2026-09-27 06:56:42.473 UTC [12473] WARNING: there is no transaction in progress
2026-09-27 06:56:42.493 UTC [12473] WARNING: there is no transaction in progress
2026-09-27 06:56:42.513 UTC [12473] WARNING: there is no transaction in progress
2026-09-27 06:56:42.581 UTC [12473] WARNING: there is no transaction in progress
2026-09-27 06:56:42.588 UTC [12473] WARNING: there is no transaction in progress
2026-09-27 06:56:42.601 UTC [12473] WARNING: there is no transaction in progress
2026-09-27 06:56:42.611 UTC [12473] WARNING: there is no transaction in progress
2026-09-27 06:56:42.620 UTC [12473] WARNING: there is no transaction in progress
2026-09-27 06:56:42.632 UTC [12473] WARNING: there is no transaction in progress
2026-09-27 06:56:42.641 UTC [12473] WARNING: there is no transaction in progress
2026-09-27 06:56:42.646 UTC [12473] WARNING: there is no transaction in progress
2026-09-27 06:56:42.651 UTC [12473] WARNING: there is no transaction in progress
2026-09-27 06:56:42.657 UTC [12473] WARNING: there is no transaction in progress
2026-09-27 06:56:43,316 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 06:56:43,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 06:56:43,549 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 06:56:43,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 06:56:43,807 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 06:56:43,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 06:56:44,085 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 06:56:44,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 06:56:44,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 06:56:44,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f892.c.gcp-upstream.internal';"
2026-09-27 06:56:44,870 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 06:56:44,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 06:56:45,127 [I] DB_PATCHES have already been applied
I20260927 06:56:45.132915 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-admin:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasktaM5HI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 06:56:45.133004 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260927 06:56:45.147063 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:43409
Sep 27, 2026 6:56:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:56:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:56:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 6:56:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 6:56:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 6:56:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 6:56:46 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 06:56:46.279405 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.864s user 0.004s sys 0.032s
Sep 27, 2026 6:57:04 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 6:57:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 6:57:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 6:57:04 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 6:57:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 6:57:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 6:57:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 6:57:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 6:57:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 6:57:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 6:57:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 6:57:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 6:57:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 6:57:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 6:57:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 6:57:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 6:57:04 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 6:57:05 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 06:57:06.208936 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 06:57:06 dist-test-slave-f892 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492226, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 06:57:06.354250 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 06:57:06.373132 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 06:57:07.098932 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 06:57:07.099026 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-kms
I20260927 06:57:07.099057 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:55811
2026-09-27 06:57:07,444 [I] DB FLAVOR :POSTGRES
2026-09-27 06:57:07,444 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 06:57:07,444 [I] Checking connection
2026-09-27 06:57:07,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 06:57:07,681 [I] connection success
2026-09-27 06:57:07,681 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 06:57:07,682 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 06:57:07,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 06:57:07,909 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 06:57:07,909 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 06:57:07,910 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 06:57:07,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55811/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 06:57:08,150 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 06:57:08.552121 388 mini_ranger_kms.cc:310] Created kms service
I20260927 06:57:08.764066 388 mini_ranger_kms.cc:326] Created kudu user
I20260927 06:57:08.807459 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 06:57:08 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492226, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 06:57:09.211098 388 mini_ranger_kms.cc:384] Added ranger policy
I20260927 06:57:09.211413 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-kms:/tmp/dist-test-tasktaM5HI/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasktaM5HI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 06:57:09.211481 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260927 06:57:09.225771 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12653
I20260927 06:57:09.225899 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:56011
Sep 27, 2026 6:57:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:57:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:57:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 6:57:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 6:57:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 6:57:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 06:57:11.462131 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.108s user 0.066s sys 0.290s
I20260927 06:57:11.462371 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 06:57:11.462533 388 mini_ranger_kms.cc:401] 127.0.97.20:56011/kms/v1/keys
Sep 27 06:57:11 dist-test-slave-f892 krb5kdc[12260](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492231, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 06:57:11 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492231, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 06:57:12 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492231, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 06:57:14 dist-test-slave-f892 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:57:14.608963 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46697
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46697
--ranger_config_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56011/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:57:14.948565 12704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:14.948835 12704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:14.948982 12704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:14.956259 12704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:57:14.956374 12704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:14.956460 12704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:14.956565 12704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:57:14.956650 12704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:57:14.966022 12704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:56011/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46697
--ranger_config_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:14.967154 12704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:14.968374 12704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:14.979925 12712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:14.979991 12709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:14.979961 12710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:14.980635 12704 server_base.cc:1034] running on GCE node
I20260927 06:57:14.981308 12704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:14.982672 12704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:14.983858 12704 hybrid_clock.cc:648] HybridClock initialized: now 1790492234983824 us; error 30 us; skew 500 ppm
I20260927 06:57:14.984202 12704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:14 dist-test-slave-f892 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:14.987092 12704 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:57:14.988379 12704 webserver.cc:460] Webserver started at http://127.0.97.62:45265/ using document root <none> and password file <none>
I20260927 06:57:14.989068 12704 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:14.989198 12704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:14.989493 12704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 06:57:14 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 06:57:15.050657 12704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "6097204e6b7b4fe49b5ebc4aac813fa0"
format_stamp: "Formatted at 2026-09-27 06:57:15 on dist-test-slave-f892"
server_key: "1379f7292eb1478609f7a7a2b1a7d834"
server_key_iv: "844085920450b36ad94efbfa7cc16c1c"
server_key_version: "kuduclusterkey@0"
I20260927 06:57:15.051553 12704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "6097204e6b7b4fe49b5ebc4aac813fa0"
format_stamp: "Formatted at 2026-09-27 06:57:15 on dist-test-slave-f892"
server_key: "1379f7292eb1478609f7a7a2b1a7d834"
server_key_iv: "844085920450b36ad94efbfa7cc16c1c"
server_key_version: "kuduclusterkey@0"
I20260927 06:57:15.056829 12704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 06:57:15.074532 12719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:15.075981 12704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 06:57:15.076192 12704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "6097204e6b7b4fe49b5ebc4aac813fa0"
format_stamp: "Formatted at 2026-09-27 06:57:15 on dist-test-slave-f892"
server_key: "1379f7292eb1478609f7a7a2b1a7d834"
server_key_iv: "844085920450b36ad94efbfa7cc16c1c"
server_key_version: "kuduclusterkey@0"
I20260927 06:57:15.076406 12704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:15.114724 12704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:15.118870 12704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:15.119203 12704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:15.138864 12704 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46697
I20260927 06:57:15.138885 12770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46697 every 8 connection(s)
I20260927 06:57:15.140211 12704 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:57:15.143784 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 12704
I20260927 06:57:15.144018 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:57:15.144282 12771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:15.144814 388 external_mini_cluster.cc:1428] Setting key 3953dd03049b6dac23dd8d889b8df21e
I20260927 06:57:15.158896 12771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0: Bootstrap starting.
Sep 27 06:57:15 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:15.162572 12771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0: Neither blocks nor log segments found. Creating new log.
I20260927 06:57:15.163658 12771 log.cc:824] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:15.167080 12771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0: No bootstrap required, opened a new log
I20260927 06:57:15.167784 12774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:15.146982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55388 (local address 127.0.97.62:46697)
0927 06:57:15.147806 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:15.147821 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:15.147867 (+ 46us) server_negotiation.cc:407] Connection header received
0927 06:57:15.149493 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:15.149525 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:15.150036 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:15.150481 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:15.151582 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:15.153536 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:15.154830 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:15.155899 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:15.156070 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:15.160537 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:15.160545 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:15.160558 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:15.160594 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:15.163555 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:15.164416 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:15.164424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:15.164432 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:15.164665 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:15.165015 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:15.165021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:15.165025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:15.165516 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:15.165785 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:15.166235 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 06:57:15.166528 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":356,"threads_started":1}
I20260927 06:57:15.175248 12771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6097204e6b7b4fe49b5ebc4aac813fa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46697 } }
I20260927 06:57:15.175590 12771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:15.175710 12771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6097204e6b7b4fe49b5ebc4aac813fa0, State: Initialized, Role: FOLLOWER
I20260927 06:57:15.176342 12771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6097204e6b7b4fe49b5ebc4aac813fa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46697 } }
I20260927 06:57:15.176620 12771 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:57:15.176764 12771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:57:15.176947 12771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:15.179085 12771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6097204e6b7b4fe49b5ebc4aac813fa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46697 } }
I20260927 06:57:15.179768 12771 leader_election.cc:302] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6097204e6b7b4fe49b5ebc4aac813fa0; no voters:
I20260927 06:57:15.180449 12771 leader_election.cc:288] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:57:15.180549 12776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:57:15.181334 12776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [term 1 LEADER]: Becoming Leader. State: Replica: 6097204e6b7b4fe49b5ebc4aac813fa0, State: Running, Role: LEADER
I20260927 06:57:15.181906 12776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6097204e6b7b4fe49b5ebc4aac813fa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46697 } }
I20260927 06:57:15.182456 12771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:57:15.185174 12778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6097204e6b7b4fe49b5ebc4aac813fa0. Latest consensus state: current_term: 1 leader_uuid: "6097204e6b7b4fe49b5ebc4aac813fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6097204e6b7b4fe49b5ebc4aac813fa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46697 } } }
I20260927 06:57:15.185596 12778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [sys.catalog]: This master's current role is: LEADER
I20260927 06:57:15.185431 12777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6097204e6b7b4fe49b5ebc4aac813fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6097204e6b7b4fe49b5ebc4aac813fa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46697 } } }
I20260927 06:57:15.185737 12777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [sys.catalog]: This master's current role is: LEADER
I20260927 06:57:15.187245 12785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:57:15.192387 12785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:57:15.192927 12771 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 06:57:15.210301 12785 catalog_manager.cc:1347] Generated new cluster ID: e07cac38ef61485da7323dda35a9b58b
I20260927 06:57:15.210440 12785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:57:15.245353 12785 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:57:15.247561 12785 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:57:15.277021 12785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0: Generated new TSK 0
I20260927 06:57:15.278337 12785 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 06:57:15.987471 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46697
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 06:57:16 dist-test-slave-f892 krb5kdc[12260](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 06:57:16.423808 12816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:16.424216 12816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:16.424449 12816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:16.436100 12816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:16.436255 12816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:16.436520 12816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:57:16.452440 12816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:56011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:16.454152 12816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:16.455834 12816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:16.478525 12827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:16.479506 12826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:16.479568 12829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:16.479900 12816 server_base.cc:1034] running on GCE node
I20260927 06:57:16.480577 12816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:16.481683 12816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:16.483003 12816 hybrid_clock.cc:648] HybridClock initialized: now 1790492236482882 us; error 105 us; skew 500 ppm
I20260927 06:57:16.483466 12816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:16 dist-test-slave-f892 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:16.490633 12816 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:57:16.494714 12816 webserver.cc:460] Webserver started at http://127.0.97.1:45549/ using document root <none> and password file <none>
I20260927 06:57:16.495566 12816 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:16.495761 12816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:16.496222 12816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 06:57:16 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 06:57:16.518555 12816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "221da13bdb3d4d9788589b13e3f7d5e1"
format_stamp: "Formatted at 2026-09-27 06:57:16 on dist-test-slave-f892"
server_key: "2820724f7664a9f45172c071d745e2f9"
server_key_iv: "10622d941b0e6a25e77dc78e93732e73"
server_key_version: "kuduclusterkey@0"
I20260927 06:57:16.519611 12816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "221da13bdb3d4d9788589b13e3f7d5e1"
format_stamp: "Formatted at 2026-09-27 06:57:16 on dist-test-slave-f892"
server_key: "2820724f7664a9f45172c071d745e2f9"
server_key_iv: "10622d941b0e6a25e77dc78e93732e73"
server_key_version: "kuduclusterkey@0"
I20260927 06:57:16.527387 12816 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260927 06:57:16.549878 12836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:16.553524 12816 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.001s
I20260927 06:57:16.553768 12816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "221da13bdb3d4d9788589b13e3f7d5e1"
format_stamp: "Formatted at 2026-09-27 06:57:16 on dist-test-slave-f892"
server_key: "2820724f7664a9f45172c071d745e2f9"
server_key_iv: "10622d941b0e6a25e77dc78e93732e73"
server_key_version: "kuduclusterkey@0"
I20260927 06:57:16.554040 12816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:16.600330 12816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:16.605494 12816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:16.605890 12816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:16.615100 12816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:57:16.616976 12816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:57:16.617098 12816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:16.617271 12816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:57:16.617341 12816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:16.686179 12816 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46283
I20260927 06:57:16.686223 12948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46283 every 8 connection(s)
I20260927 06:57:16.687963 12816 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:57:16.689646 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 12816
I20260927 06:57:16.689852 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:57:16.690712 388 external_mini_cluster.cc:1428] Setting key 020a58655c4e83de7b58ea5bfd6fc8d3
Sep 27 06:57:16 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 06:57:16.747223 12952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:16.703496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46697 (local address 127.0.97.1:49503)
0927 06:57:16.719537 (+ 16041us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:16.719599 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:16.722318 (+ 2719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:16.725020 (+ 2702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:16.725036 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:16.725716 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:16.726587 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:16.726618 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:16.728126 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:16.728134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:16.730380 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:16.730398 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:16.732139 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:16.732161 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:16.732461 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:16.733289 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:16.733327 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:16.736270 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:16.742368 (+ 6098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:16.742378 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:16.742396 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:16.742974 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:16.743453 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:16.743457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:16.743461 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:16.743747 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:16.744443 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:16.744452 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:16.744890 (+ 438us) client_negotiation.cc:769] Sending connection context
0927 06:57:16.745200 (+ 310us) client_negotiation.cc:240] Negotiation successful
0927 06:57:16.745727 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15712,"thread_start_us":15480,"threads_started":1}
I20260927 06:57:16.748499 12953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:16.704085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49503 (local address 127.0.97.62:46697)
0927 06:57:16.724394 (+ 20309us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:16.724402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:16.724425 (+ 23us) server_negotiation.cc:407] Connection header received
0927 06:57:16.724496 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:16.724501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:16.724599 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:16.724735 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:16.726810 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:16.727970 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:16.731235 (+ 3265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:16.731882 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:16.732014 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:16.738185 (+ 6171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:16.738194 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:16.738199 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:16.738242 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:16.742206 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:16.743140 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:16.743146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:16.743150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:16.743318 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:16.743902 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:16.743907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:16.743910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:16.744291 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:16.747727 (+ 3436us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:16.747885 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 06:57:16.748113 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20162,"thread_start_us":20767,"threads_started":1}
I20260927 06:57:16.749910 12949 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46697
I20260927 06:57:16.750254 12949 heartbeater.cc:464] Registering TS with master...
I20260927 06:57:16.751046 12949 heartbeater.cc:511] Master 127.0.97.62:46697 requested a full tablet report, sending...
I20260927 06:57:16.753438 12736 ts_manager.cc:194] Registered new tserver with Master: 221da13bdb3d4d9788589b13e3f7d5e1 (127.0.97.1:46283)
I20260927 06:57:16.755383 12736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 06:57:16.779206 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46697
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 06:57:16 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 06:57:17 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 06:57:17.224614 12955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:17.224964 12955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:17.225128 12955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:17.236385 12955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:17.236583 12955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:17.236815 12955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:57:17.251430 12955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:56011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:17.252959 12955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:17.254527 12955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:17.288074 12962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:17.296139 12966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:17.296545 12955 server_base.cc:1034] running on GCE node
W20260927 06:57:17.296672 12963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:17.298923 12955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:17.314280 12955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:17.315510 12955 hybrid_clock.cc:648] HybridClock initialized: now 1790492237315454 us; error 60 us; skew 500 ppm
I20260927 06:57:17.315909 12955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:17 dist-test-slave-f892 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:17.320168 12955 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:57:17.323271 12955 webserver.cc:460] Webserver started at http://127.0.97.2:36939/ using document root <none> and password file <none>
I20260927 06:57:17.323987 12955 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:17.324137 12955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:17.324460 12955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 06:57:17 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 06:57:17.339648 12955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "be57c46ae3394c41a516578e130323c0"
format_stamp: "Formatted at 2026-09-27 06:57:17 on dist-test-slave-f892"
server_key: "310ad893e04ff95e874674be9d8646ed"
server_key_iv: "9a750f87ca48a90e6e48af5f36082e65"
server_key_version: "kuduclusterkey@0"
I20260927 06:57:17.340672 12955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "be57c46ae3394c41a516578e130323c0"
format_stamp: "Formatted at 2026-09-27 06:57:17 on dist-test-slave-f892"
server_key: "310ad893e04ff95e874674be9d8646ed"
server_key_iv: "9a750f87ca48a90e6e48af5f36082e65"
server_key_version: "kuduclusterkey@0"
I20260927 06:57:17.346354 12955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 06:57:17.374979 12975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:17.376694 12955 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260927 06:57:17.376916 12955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "be57c46ae3394c41a516578e130323c0"
format_stamp: "Formatted at 2026-09-27 06:57:17 on dist-test-slave-f892"
server_key: "310ad893e04ff95e874674be9d8646ed"
server_key_iv: "9a750f87ca48a90e6e48af5f36082e65"
server_key_version: "kuduclusterkey@0"
I20260927 06:57:17.377203 12955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:17.418439 12955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:17.422648 12955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:17.422956 12955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:17.423967 12955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:57:17.425331 12955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:57:17.425401 12955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:17.425535 12955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:57:17.425585 12955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:17.459962 12955 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35511
I20260927 06:57:17.460041 13091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35511 every 8 connection(s)
I20260927 06:57:17.461153 12955 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:57:17.466571 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 12955
I20260927 06:57:17.466747 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:57:17.467135 388 external_mini_cluster.cc:1428] Setting key 1b20f2b9ca65d374ad6c5e94b7ac6cc7
Sep 27 06:57:17 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:17.482838 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:17.463810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39813 (local address 127.0.97.62:46697)
0927 06:57:17.464306 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:17.464312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:17.465219 (+ 907us) server_negotiation.cc:407] Connection header received
0927 06:57:17.467632 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:17.467637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:17.467733 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:17.467855 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:17.469577 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:17.470513 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:17.472130 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:17.472511 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:17.472644 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:17.475799 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:17.475803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:17.475806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:17.475836 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:17.479025 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:17.479871 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:17.479880 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:17.479886 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:17.480057 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:17.480591 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:17.480599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:17.480605 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:17.480987 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:17.481147 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:17.482044 (+ 897us) server_negotiation.cc:299] Negotiation successful
0927 06:57:17.482310 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":241,"threads_started":1}
I20260927 06:57:17.483887 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:17.464296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46697 (local address 127.0.97.2:39813)
0927 06:57:17.465046 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:17.465105 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:17.467306 (+ 2201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:17.468074 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:17.468082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:17.468610 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:17.469364 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:17.469390 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:17.470759 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:17.470764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:17.471981 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:17.471992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:17.472757 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:17.472765 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:17.472997 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:17.473631 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:17.473651 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:17.475598 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:17.479160 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:17.479168 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:17.479181 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:17.479740 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:17.480167 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:17.480171 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:17.480174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:17.480444 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:17.481119 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:17.481126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:17.481547 (+ 421us) client_negotiation.cc:769] Sending connection context
0927 06:57:17.481955 (+ 408us) client_negotiation.cc:240] Negotiation successful
0927 06:57:17.482475 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":299,"threads_started":1}
I20260927 06:57:17.485340 13092 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46697
I20260927 06:57:17.485647 13092 heartbeater.cc:464] Registering TS with master...
I20260927 06:57:17.486456 13092 heartbeater.cc:511] Master 127.0.97.62:46697 requested a full tablet report, sending...
I20260927 06:57:17.488031 12736 ts_manager.cc:194] Registered new tserver with Master: be57c46ae3394c41a516578e130323c0 (127.0.97.2:35511)
I20260927 06:57:17.489311 12736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39813
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 06:57:17.521639 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46697
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:57:17.758945 12949 heartbeater.cc:503] Master 127.0.97.62:46697 was elected leader, sending a full tablet report...
W20260927 06:57:17.836506 13098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:17.836783 13098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:17.836942 13098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:17.843707 13098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:17.843842 13098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:17.844020 13098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:57:17.854703 13098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:56011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:17.856086 13098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:17.857304 13098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:17.870188 13103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:17.870139 13104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:17.870223 13106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:17.870905 13098 server_base.cc:1034] running on GCE node
I20260927 06:57:17.871503 13098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:17.872388 13098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:17.873610 13098 hybrid_clock.cc:648] HybridClock initialized: now 1790492237873561 us; error 55 us; skew 500 ppm
I20260927 06:57:17.873900 13098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:17 dist-test-slave-f892 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:17.876812 13098 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:57:17.878125 13098 webserver.cc:460] Webserver started at http://127.0.97.3:43057/ using document root <none> and password file <none>
I20260927 06:57:17.878717 13098 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:17.878851 13098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:17.879158 13098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 06:57:17 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 06:57:17.895664 13098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "d5a14e1135ae418e84b127f1f1cb370a"
format_stamp: "Formatted at 2026-09-27 06:57:17 on dist-test-slave-f892"
server_key: "0d00f4873f7097b56ec72b97f30f891e"
server_key_iv: "c8cb40e745f5b90154efe44a70faa12e"
server_key_version: "kuduclusterkey@0"
I20260927 06:57:17.896682 13098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d5a14e1135ae418e84b127f1f1cb370a"
format_stamp: "Formatted at 2026-09-27 06:57:17 on dist-test-slave-f892"
server_key: "0d00f4873f7097b56ec72b97f30f891e"
server_key_iv: "c8cb40e745f5b90154efe44a70faa12e"
server_key_version: "kuduclusterkey@0"
I20260927 06:57:17.903203 13098 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 06:57:17.919248 13113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:17.920642 13098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 06:57:17.920776 13098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "d5a14e1135ae418e84b127f1f1cb370a"
format_stamp: "Formatted at 2026-09-27 06:57:17 on dist-test-slave-f892"
server_key: "0d00f4873f7097b56ec72b97f30f891e"
server_key_iv: "c8cb40e745f5b90154efe44a70faa12e"
server_key_version: "kuduclusterkey@0"
I20260927 06:57:17.920977 13098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:17.946861 13098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:17.950518 13098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:17.950748 13098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:17.951584 13098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:57:17.952791 13098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:57:17.952858 13098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:17.952935 13098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:57:17.952977 13098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:17.984643 13098 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43189
I20260927 06:57:17.984683 13225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43189 every 8 connection(s)
I20260927 06:57:17.985801 13098 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:57:17.993901 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 13098
I20260927 06:57:17.994123 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:57:17.994534 388 external_mini_cluster.cc:1428] Setting key 272adead155abd9f44ed01bdd925a334
Sep 27 06:57:17 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:18.003834 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:17.988153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42645 (local address 127.0.97.62:46697)
0927 06:57:17.988627 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:17.988632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:17.989502 (+ 870us) server_negotiation.cc:407] Connection header received
0927 06:57:17.991153 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:17.991157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:17.991236 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:17.991359 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:17.992710 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:17.993404 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:17.995001 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:17.995604 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:17.995727 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:17.999172 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:17.999176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:17.999179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:17.999209 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:18.001429 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:18.002113 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:18.002117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:18.002119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:18.002233 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:18.002559 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:18.002562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:18.002563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:18.002810 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:18.002924 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:18.003448 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 06:57:18.003589 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":220,"threads_started":1}
I20260927 06:57:18.004665 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:17.988716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46697 (local address 127.0.97.3:42645)
0927 06:57:17.989361 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:17.989400 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:17.990916 (+ 1516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:17.991586 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:17.991594 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:17.992034 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:17.992554 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:17.992569 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:17.993548 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:17.993558 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:17.994863 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:17.994873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:17.995837 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:17.995843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:17.996031 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:17.996724 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:17.996752 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:17.998985 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:18.001543 (+ 2558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:18.001549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:18.001558 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:18.001959 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:18.002320 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:18.002323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:18.002325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:18.002470 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:18.002911 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:18.002916 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:18.003179 (+ 263us) client_negotiation.cc:769] Sending connection context
0927 06:57:18.003369 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 06:57:18.003679 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":273,"threads_started":1}
I20260927 06:57:18.005937 13226 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46697
I20260927 06:57:18.006213 13226 heartbeater.cc:464] Registering TS with master...
I20260927 06:57:18.006780 13226 heartbeater.cc:511] Master 127.0.97.62:46697 requested a full tablet report, sending...
I20260927 06:57:18.007988 12736 ts_manager.cc:194] Registered new tserver with Master: d5a14e1135ae418e84b127f1f1cb370a (127.0.97.3:43189)
I20260927 06:57:18.008960 12736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42645
I20260927 06:57:18.009691 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:57:18.011303 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 12816
Sep 27 06:57:18 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:57:18.027220 13231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:18.012227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.0.97.1:46283)
0927 06:57:18.012818 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:18.012824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:18.012845 (+ 21us) server_negotiation.cc:407] Connection header received
0927 06:57:18.012894 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:18.012898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:18.013060 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:18.013170 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:18.014561 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:18.015500 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:18.016982 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:18.017501 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:18.017619 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:18.021741 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:18.021752 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:18.021765 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:18.021798 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:18.024729 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:18.025456 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:18.025463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:18.025465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:18.025621 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:18.025933 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:18.025936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:18.025938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:18.026286 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:18.026512 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:18.026775 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 06:57:18.026960 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":309,"threads_started":1}
I20260927 06:57:18.028381 12944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:18.109421 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 12816
I20260927 06:57:18.124666 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 12955
Sep 27 06:57:18 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:57:18.140497 13233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:18.125592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.0.97.2:35511)
0927 06:57:18.126160 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:18.126169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:18.126202 (+ 33us) server_negotiation.cc:407] Connection header received
0927 06:57:18.126273 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:18.126279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:18.126479 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:18.126608 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:18.127736 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:18.128702 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:18.129979 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:18.130598 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:18.130691 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:18.134855 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:18.134866 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:18.134878 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:18.134909 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:18.137646 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:18.138424 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:18.138434 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:18.138437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:18.138591 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:18.138945 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:18.138949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:18.138953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:18.139393 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:18.139663 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:18.139852 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 06:57:18.140089 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":287,"threads_started":1}
I20260927 06:57:18.141770 13087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:18.221627 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 12955
I20260927 06:57:18.237314 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 13098
Sep 27 06:57:18 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 06:57:18.252393 13235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:18.238184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58464 (local address 127.0.97.3:43189)
0927 06:57:18.238818 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:18.238825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:18.238852 (+ 27us) server_negotiation.cc:407] Connection header received
0927 06:57:18.238912 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:18.238918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:18.239121 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:18.239248 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:18.240353 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:18.241308 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:18.242596 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:18.243156 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:18.243252 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:18.247377 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:18.247387 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:18.247399 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:18.247425 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:18.250133 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:18.250740 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:18.250747 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:18.250749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:18.250874 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:18.251196 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:18.251200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:18.251201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:18.251490 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:18.251734 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:18.251946 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 06:57:18.252122 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":287,"threads_started":1}
I20260927 06:57:18.253602 13221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:18.333062 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 13098
I20260927 06:57:18.348260 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 12704
I20260927 06:57:18.349370 12766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:18.453445 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 12704
I20260927 06:57:18.474974 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46697
--webserver_interface=127.0.97.62
--webserver_port=45265
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46697
--ranger_config_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56011/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:56011/kms with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:57:18.781466 13238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:18.781801 13238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:18.781972 13238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:18.788940 13238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:57:18.789076 13238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:18.789178 13238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:18.789280 13238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:57:18.789369 13238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:57:18.799176 13238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:56011/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46697
--ranger_config_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:18.800375 13238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:18.801628 13238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:18.812783 13243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:18.812808 13244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:18.812767 13246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:18.813889 13238 server_base.cc:1034] running on GCE node
I20260927 06:57:18.814646 13238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:18.815912 13238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:18.817155 13238 hybrid_clock.cc:648] HybridClock initialized: now 1790492238817082 us; error 81 us; skew 500 ppm
I20260927 06:57:18.817544 13238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:18 dist-test-slave-f892 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:18.821139 13238 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:57:18.822655 13238 webserver.cc:460] Webserver started at http://127.0.97.62:45265/ using document root <none> and password file <none>
I20260927 06:57:18.823243 13238 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:18.823347 13238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 06:57:18 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 06:57:18.888731 13238 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 06:57:18.891475 13254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:18.892765 13238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 06:57:18.892911 13238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "6097204e6b7b4fe49b5ebc4aac813fa0"
format_stamp: "Formatted at 2026-09-27 06:57:15 on dist-test-slave-f892"
server_key: "1379f7292eb1478609f7a7a2b1a7d834"
server_key_iv: "844085920450b36ad94efbfa7cc16c1c"
server_key_version: "kuduclusterkey@0"
I20260927 06:57:18.893790 13238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:18.933586 13238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:18.937239 13238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:18.937462 13238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:18.954843 13238 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46697
I20260927 06:57:18.954864 13305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46697 every 8 connection(s)
I20260927 06:57:18.956192 13238 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:57:18.962582 13306 sys_catalog.cc:263] Verifying existing consensus state
I20260927 06:57:18.965554 13306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0: Bootstrap starting.
I20260927 06:57:18.966063 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 13238
I20260927 06:57:18.966852 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:46283
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46697
--webserver_port=45549
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:56011/kms with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:57:18.983405 13306 log.cc:824] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:18.991657 13306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 06:57:18.992168 13306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0: Bootstrap complete.
I20260927 06:57:19.001020 13306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6097204e6b7b4fe49b5ebc4aac813fa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46697 } }
I20260927 06:57:19.001847 13306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6097204e6b7b4fe49b5ebc4aac813fa0, State: Initialized, Role: FOLLOWER
I20260927 06:57:19.002446 13306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6097204e6b7b4fe49b5ebc4aac813fa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46697 } }
I20260927 06:57:19.002674 13306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:57:19.002791 13306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:57:19.002972 13306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 06:57:19.006515 13306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6097204e6b7b4fe49b5ebc4aac813fa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46697 } }
I20260927 06:57:19.007009 13306 leader_election.cc:302] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6097204e6b7b4fe49b5ebc4aac813fa0; no voters:
I20260927 06:57:19.007553 13306 leader_election.cc:288] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 06:57:19.007750 13310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 06:57:19.008996 13310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [term 2 LEADER]: Becoming Leader. State: Replica: 6097204e6b7b4fe49b5ebc4aac813fa0, State: Running, Role: LEADER
I20260927 06:57:19.009692 13306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:57:19.009639 13310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6097204e6b7b4fe49b5ebc4aac813fa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46697 } }
I20260927 06:57:19.012499 13311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6097204e6b7b4fe49b5ebc4aac813fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6097204e6b7b4fe49b5ebc4aac813fa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46697 } } }
I20260927 06:57:19.012496 13312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6097204e6b7b4fe49b5ebc4aac813fa0. Latest consensus state: current_term: 2 leader_uuid: "6097204e6b7b4fe49b5ebc4aac813fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6097204e6b7b4fe49b5ebc4aac813fa0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46697 } } }
I20260927 06:57:19.012955 13311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [sys.catalog]: This master's current role is: LEADER
I20260927 06:57:19.012982 13312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0 [sys.catalog]: This master's current role is: LEADER
I20260927 06:57:19.013832 13315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:57:19.027901 13315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:57:19.029312 13315 catalog_manager.cc:1259] Loaded cluster ID: e07cac38ef61485da7323dda35a9b58b
I20260927 06:57:19.029409 13315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:57:19.033278 13315 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:57:19.034660 13315 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0: Loaded TSK: 0
I20260927 06:57:19.036098 13315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:57:19.043062 13306 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260927 06:57:19.396168 13308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:19.396505 13308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:19.396745 13308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:19.408806 13308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:19.408941 13308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:19.409173 13308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:57:19.424692 13308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:56011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:46283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45549
--tserver_master_addrs=127.0.97.62:46697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:19.426297 13308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:19.427942 13308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:19.458952 13344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:19.460462 13343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:19.494424 13308 server_base.cc:1034] running on GCE node
W20260927 06:57:19.496691 13346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:19.497576 13308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:19.506333 13308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:19.509763 13308 hybrid_clock.cc:648] HybridClock initialized: now 1790492239509649 us; error 104 us; skew 500 ppm
I20260927 06:57:19.510263 13308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:19 dist-test-slave-f892 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:19.514793 13308 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:57:19.529503 13308 webserver.cc:460] Webserver started at http://127.0.97.1:45549/ using document root <none> and password file <none>
I20260927 06:57:19.530546 13308 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:19.530714 13308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 06:57:19 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 06:57:19.561611 13308 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 06:57:19.568847 13355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:19.570565 13308 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.002s
I20260927 06:57:19.570767 13308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "221da13bdb3d4d9788589b13e3f7d5e1"
format_stamp: "Formatted at 2026-09-27 06:57:16 on dist-test-slave-f892"
server_key: "2820724f7664a9f45172c071d745e2f9"
server_key_iv: "10622d941b0e6a25e77dc78e93732e73"
server_key_version: "kuduclusterkey@0"
I20260927 06:57:19.571841 13308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:19.623270 13308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:19.627965 13308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:19.628345 13308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:19.629678 13308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:57:19.631305 13308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:57:19.631419 13308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:19.631543 13308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:57:19.631654 13308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:19.712831 13308 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46283
I20260927 06:57:19.712883 13467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46283 every 8 connection(s)
I20260927 06:57:19.714375 13308 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:57:19.720585 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 13308
I20260927 06:57:19.721331 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35511
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46697
--webserver_port=36939
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:56011/kms with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 06:57:19 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:19.743708 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:19.718445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46697 (local address 127.0.97.1:39755)
0927 06:57:19.719396 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:19.719467 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:19.723146 (+ 3679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:19.725599 (+ 2453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:19.725609 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:19.726236 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:19.726978 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:19.727003 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:19.729309 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:19.729314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:19.731284 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:19.731307 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:19.732368 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:19.732376 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:19.732645 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:19.733330 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:19.733352 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:19.735550 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:19.739394 (+ 3844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:19.739402 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:19.739417 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:19.739830 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:19.740271 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:19.740276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:19.740280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:19.740451 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:19.741259 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:19.741268 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:19.741573 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 06:57:19.741832 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 06:57:19.742392 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":344,"threads_started":1}
I20260927 06:57:19.743697 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:19.718720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39755 (local address 127.0.97.62:46697)
0927 06:57:19.720190 (+ 1470us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:19.720208 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:19.720261 (+ 53us) server_negotiation.cc:407] Connection header received
0927 06:57:19.724001 (+ 3740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:19.724020 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:19.724532 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:19.724927 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:19.727221 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:19.729109 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:19.731440 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:19.732080 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:19.732286 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:19.735832 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:19.735842 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:19.735860 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:19.735911 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:19.739254 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:19.739979 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:19.739990 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:19.739998 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:19.740151 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:19.740568 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:19.740577 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:19.740579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:19.741106 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:19.741328 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:19.742077 (+ 749us) server_negotiation.cc:299] Negotiation successful
0927 06:57:19.742447 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1214,"thread_start_us":955,"threads_started":1}
I20260927 06:57:19.746811 13468 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46697
I20260927 06:57:19.747120 13468 heartbeater.cc:464] Registering TS with master...
I20260927 06:57:19.747761 13468 heartbeater.cc:511] Master 127.0.97.62:46697 requested a full tablet report, sending...
W20260927 06:57:19.750365 13468 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46697 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 06:57:19.750587 13468 heartbeater.cc:464] Registering TS with master...
I20260927 06:57:19.750830 13468 heartbeater.cc:511] Master 127.0.97.62:46697 requested a full tablet report, sending...
I20260927 06:57:19.751930 13468 heartbeater.cc:464] Registering TS with master...
I20260927 06:57:19.752178 13468 heartbeater.cc:511] Master 127.0.97.62:46697 requested a full tablet report, sending...
W20260927 06:57:19.753120 13468 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 06:57:20.148114 13472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:20.148469 13472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:20.148609 13472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:20.160089 13472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:20.160243 13472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:20.160439 13472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:57:20.175832 13472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:56011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36939
--tserver_master_addrs=127.0.97.62:46697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:20.177436 13472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:20.179070 13472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:20.193912 13488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:20.195000 13489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:20.195024 13491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:20.196023 13472 server_base.cc:1034] running on GCE node
I20260927 06:57:20.196619 13472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:20.197722 13472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Sep 27 06:57:20 dist-test-slave-f892 krb5kdc[12260](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:20.198937 13472 hybrid_clock.cc:648] HybridClock initialized: now 1790492240198869 us; error 60 us; skew 500 ppm
I20260927 06:57:20.199278 13472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:20 dist-test-slave-f892 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:20.202399 13472 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:57:20.204998 13472 webserver.cc:460] Webserver started at http://127.0.97.2:36939/ using document root <none> and password file <none>
I20260927 06:57:20.205852 13472 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:20.206017 13472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 06:57:20 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 06:57:20.234727 13472 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.000s
I20260927 06:57:20.237658 13499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:20.239032 13472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 06:57:20.239202 13472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "be57c46ae3394c41a516578e130323c0"
format_stamp: "Formatted at 2026-09-27 06:57:17 on dist-test-slave-f892"
server_key: "310ad893e04ff95e874674be9d8646ed"
server_key_iv: "9a750f87ca48a90e6e48af5f36082e65"
server_key_version: "kuduclusterkey@0"
I20260927 06:57:20.240118 13472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:20.264622 13472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:20.269538 13472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:20.269829 13472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:20.270951 13472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:57:20.272893 13472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:57:20.273003 13472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:20.273106 13472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:57:20.273183 13472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:20.326690 13472 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35511
I20260927 06:57:20.326870 13612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35511 every 8 connection(s)
I20260927 06:57:20.328384 13472 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:57:20.331845 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 13472
I20260927 06:57:20.332605 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43189
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46697
--webserver_port=43057
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:56011/kms with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 06:57:20 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:20.376847 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:20.331256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45527 (local address 127.0.97.62:46697)
0927 06:57:20.354217 (+ 22961us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:20.354224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:20.354243 (+ 19us) server_negotiation.cc:407] Connection header received
0927 06:57:20.354307 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:20.354312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:20.354396 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:20.354496 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:20.356579 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:20.357860 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:20.360621 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:20.361383 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:20.361496 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:20.369733 (+ 8237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:20.369741 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:20.369746 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:20.369785 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:20.373323 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:20.374254 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:20.374260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:20.374264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:20.374391 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:20.374840 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:20.374844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:20.374848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:20.375187 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:20.375306 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:20.376217 (+ 911us) server_negotiation.cc:299] Negotiation successful
0927 06:57:20.376445 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22821,"thread_start_us":22695,"threads_started":1}
I20260927 06:57:20.379441 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:20.332036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46697 (local address 127.0.97.2:45527)
0927 06:57:20.350587 (+ 18551us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:20.350638 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:20.353416 (+ 2778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:20.354898 (+ 1482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:20.354909 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:20.355544 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:20.356374 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:20.356405 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:20.358115 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:20.358124 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:20.360402 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:20.360446 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:20.361851 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:20.361860 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:20.362246 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:20.366743 (+ 4497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:20.366774 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:20.369467 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:20.373513 (+ 4046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:20.373522 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:20.373540 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:20.374127 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:20.374516 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:20.374521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:20.374524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:20.374726 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:20.375398 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:20.375409 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:20.375802 (+ 393us) client_negotiation.cc:769] Sending connection context
0927 06:57:20.377352 (+ 1550us) client_negotiation.cc:240] Negotiation successful
0927 06:57:20.377976 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18215,"thread_start_us":21983,"threads_started":1}
I20260927 06:57:20.381219 13613 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46697
I20260927 06:57:20.381500 13613 heartbeater.cc:464] Registering TS with master...
I20260927 06:57:20.382339 13613 heartbeater.cc:511] Master 127.0.97.62:46697 requested a full tablet report, sending...
I20260927 06:57:20.384209 13271 ts_manager.cc:194] Registered new tserver with Master: be57c46ae3394c41a516578e130323c0 (127.0.97.2:35511)
I20260927 06:57:20.386098 13271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45527
Sep 27 06:57:20 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 06:57:20.753245 13468 heartbeater.cc:464] Registering TS with master...
I20260927 06:57:20.753721 13468 heartbeater.cc:511] Master 127.0.97.62:46697 requested a full tablet report, sending...
I20260927 06:57:20.755168 13271 ts_manager.cc:194] Registered new tserver with Master: 221da13bdb3d4d9788589b13e3f7d5e1 (127.0.97.1:46283)
I20260927 06:57:20.756362 13271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39755
W20260927 06:57:20.806129 13617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:20.806604 13617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:20.806800 13617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:20.818143 13617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:20.818365 13617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:20.818578 13617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:57:20.833818 13617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:56011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43057
--tserver_master_addrs=127.0.97.62:46697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:20.835717 13617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:20.837337 13617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:20.872704 13623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:20.874384 13622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:20.901914 13617 server_base.cc:1034] running on GCE node
W20260927 06:57:20.902701 13625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:20.903545 13617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:20.909996 13617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:20.911309 13617 hybrid_clock.cc:648] HybridClock initialized: now 1790492240911244 us; error 58 us; skew 500 ppm
I20260927 06:57:20.911755 13617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:20 dist-test-slave-f892 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:20.919477 13617 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:57:20.954414 13617 webserver.cc:460] Webserver started at http://127.0.97.3:43057/ using document root <none> and password file <none>
I20260927 06:57:20.955665 13617 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:20.955896 13617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 06:57:20 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 06:57:20.994886 13617 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.002s
I20260927 06:57:21.010656 13634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:21.014253 13617 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.009s sys 0.008s
I20260927 06:57:21.014452 13617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "d5a14e1135ae418e84b127f1f1cb370a"
format_stamp: "Formatted at 2026-09-27 06:57:17 on dist-test-slave-f892"
server_key: "0d00f4873f7097b56ec72b97f30f891e"
server_key_iv: "c8cb40e745f5b90154efe44a70faa12e"
server_key_version: "kuduclusterkey@0"
I20260927 06:57:21.015823 13617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 27 06:57:21 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 06:57:21.064157 13617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:21.069156 13617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:21.069439 13617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:21.086447 13617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:57:21.094450 13617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:57:21.094579 13617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:21.094696 13617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:57:21.094789 13617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:21.319437 13617 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43189
I20260927 06:57:21.319476 13746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43189 every 8 connection(s)
I20260927 06:57:21.320886 13617 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:57:21.329125 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 13617
Sep 27 06:57:21 dist-test-slave-f892 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:21.371724 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:21.337255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42400 (local address 127.0.97.62:46697)
0927 06:57:21.354442 (+ 17187us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:21.354447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:21.354461 (+ 14us) server_negotiation.cc:407] Connection header received
0927 06:57:21.354493 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:21.354499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:21.354572 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:21.354675 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:21.356587 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.357706 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.359593 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.360238 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.360376 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:21.363802 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:21.363809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:21.363814 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:21.363857 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:21.367410 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:21.368215 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:21.368221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:21.368223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:21.368369 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:21.368829 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:21.368834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:21.368837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:21.369181 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:21.369421 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:21.371108 (+ 1687us) server_negotiation.cc:299] Negotiation successful
0927 06:57:21.371326 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17062,"thread_start_us":16928,"threads_started":1}
I20260927 06:57:21.379065 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:21.325630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45033 (local address 127.0.97.62:46697)
0927 06:57:21.337285 (+ 11655us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:21.337292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:21.346315 (+ 9023us) server_negotiation.cc:407] Connection header received
0927 06:57:21.354356 (+ 8041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:21.354361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:21.354468 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:21.356617 (+ 2149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:21.359429 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.361652 (+ 2223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.364230 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.364948 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.365065 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:21.370377 (+ 5312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:21.370383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:21.370387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:21.370433 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:21.375311 (+ 4878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:21.376261 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:21.376269 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:21.376274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:21.376438 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:21.377001 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:21.377006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:21.377010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:21.377333 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:21.378581 (+ 1248us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:21.378710 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 06:57:21.378841 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11532,"thread_start_us":11404,"threads_started":1}
I20260927 06:57:21.379779 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:21.324932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46697 (local address 127.0.97.3:45033)
0927 06:57:21.345535 (+ 20603us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:21.345581 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:21.347925 (+ 2344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:21.356447 (+ 8522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:21.356459 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:21.358427 (+ 1968us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:21.359184 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:21.359213 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.361837 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.361842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:21.364045 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:21.364060 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.365478 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.365487 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:21.365866 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:21.366739 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:21.366780 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:21.370107 (+ 3327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:21.375453 (+ 5346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:21.375462 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:21.375479 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:21.376105 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:21.376600 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:21.376607 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:21.376611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:21.376850 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:21.377517 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:21.377525 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:21.377906 (+ 381us) client_negotiation.cc:769] Sending connection context
0927 06:57:21.378231 (+ 325us) client_negotiation.cc:240] Negotiation successful
0927 06:57:21.378732 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19484,"thread_start_us":19572,"threads_started":1}
I20260927 06:57:21.381002 13747 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46697
I20260927 06:57:21.381266 13747 heartbeater.cc:464] Registering TS with master...
I20260927 06:57:21.381915 13747 heartbeater.cc:511] Master 127.0.97.62:46697 requested a full tablet report, sending...
I20260927 06:57:21.383320 13271 ts_manager.cc:194] Registered new tserver with Master: d5a14e1135ae418e84b127f1f1cb370a (127.0.97.3:43189)
I20260927 06:57:21.384438 13271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45033
I20260927 06:57:21.386714 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:57:21.391175 13613 heartbeater.cc:503] Master 127.0.97.62:46697 was elected leader, sending a full tablet report...
I20260927 06:57:21.402132 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:21.389710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42406 (local address 127.0.97.62:46697)
0927 06:57:21.389879 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:21.389886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:21.390209 (+ 323us) server_negotiation.cc:407] Connection header received
0927 06:57:21.390346 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:21.390353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:21.390448 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:21.390539 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:21.392132 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.392972 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.394352 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.394761 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.394855 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:21.397129 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:21.397133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:21.397150 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:21.397187 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:21.399393 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:21.399950 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:21.399975 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:21.399981 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:21.400151 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:21.400543 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:21.400548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:21.400552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:21.400928 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:21.401185 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:21.401426 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 06:57:21.401706 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 06:57:21.407645 13271 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:57:21.414919 13271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:57:21.446547 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:21.427962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35511 (local address 127.0.0.1:44074)
0927 06:57:21.434650 (+ 6688us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:21.434686 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:21.434870 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:21.439058 (+ 4188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:21.439066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:21.439100 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:21.439617 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:21.439632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.441493 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.441504 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:21.443652 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:21.443666 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.445587 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.445593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:21.445780 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:21.445842 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 06:57:21.445943 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 06:57:21.446164 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6437,"thread_start_us":9752,"threads_started":1}
I20260927 06:57:21.446655 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:21.427962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46283 (local address 127.0.0.1:45228)
0927 06:57:21.437763 (+ 9801us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:21.437781 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:21.437928 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:21.438842 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:21.438848 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:21.438879 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:21.439458 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:21.439471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.441293 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.441298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:21.443818 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:21.443831 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.445893 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.445898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:21.446091 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:21.446119 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 06:57:21.446201 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 06:57:21.446321 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9663,"mutex_wait_us":297,"thread_start_us":10796,"threads_started":1}
I20260927 06:57:21.446945 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:21.428068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45228 (local address 127.0.97.1:46283)
0927 06:57:21.434238 (+ 6170us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:21.434251 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:21.438080 (+ 3829us) server_negotiation.cc:407] Connection header received
0927 06:57:21.438190 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:21.438198 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:21.438487 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:21.438691 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:21.439695 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.441135 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.443967 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.445363 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.445504 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:21.445744 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:21.446454 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 06:57:21.446637 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5937,"thread_start_us":5748,"threads_started":1}
I20260927 06:57:21.447535 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:21.429518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.0.97.3:43189)
0927 06:57:21.437679 (+ 8161us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:21.437691 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:21.439808 (+ 2117us) server_negotiation.cc:407] Connection header received
0927 06:57:21.439911 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:21.439917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:21.440104 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:21.440277 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:21.440783 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.442298 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.444731 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.445765 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.445881 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:21.446111 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:21.447199 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0927 06:57:21.447319 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7948,"thread_start_us":7739,"threads_started":1}
I20260927 06:57:21.447708 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:21.428976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44074 (local address 127.0.97.2:35511)
0927 06:57:21.438325 (+ 9349us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:21.438349 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:21.438380 (+ 31us) server_negotiation.cc:407] Connection header received
0927 06:57:21.438515 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:21.438526 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:21.438798 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:21.439732 (+ 934us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:21.439947 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.441150 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.443855 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.445363 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.446977 (+ 1614us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:21.447224 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:21.447289 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 06:57:21.447434 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9096,"thread_start_us":8006,"threads_started":1}
I20260927 06:57:21.448122 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:21.428468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43189 (local address 127.0.0.1:45612)
0927 06:57:21.439524 (+ 11056us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:21.439545 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:21.439681 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:21.440286 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:21.440291 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:21.440314 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:21.440599 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:21.440609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.442459 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.442466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:21.444562 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:21.444573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.446260 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.446266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:21.446454 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:21.446486 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 06:57:21.446573 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 06:57:21.447803 (+ 1230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10920,"mutex_wait_us":1,"thread_start_us":5794,"threads_started":1}
I20260927 06:57:21.451108 13682 tablet_service.cc:1467] Processing CreateTablet for tablet 223c30db1d084f1c9b9f4333c3fd2bf1 (DEFAULT_TABLE table=test-table [id=3ee2b89f1897442f849a563885934aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:57:21.451117 13548 tablet_service.cc:1467] Processing CreateTablet for tablet 223c30db1d084f1c9b9f4333c3fd2bf1 (DEFAULT_TABLE table=test-table [id=3ee2b89f1897442f849a563885934aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:57:21.451109 13403 tablet_service.cc:1467] Processing CreateTablet for tablet 223c30db1d084f1c9b9f4333c3fd2bf1 (DEFAULT_TABLE table=test-table [id=3ee2b89f1897442f849a563885934aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:57:21.452328 13682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 223c30db1d084f1c9b9f4333c3fd2bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:21.452401 13548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 223c30db1d084f1c9b9f4333c3fd2bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:21.452419 13403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 223c30db1d084f1c9b9f4333c3fd2bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:21.464432 13775 tablet_bootstrap.cc:492] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a: Bootstrap starting.
I20260927 06:57:21.466676 13777 tablet_bootstrap.cc:492] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1: Bootstrap starting.
I20260927 06:57:21.467844 13775 tablet_bootstrap.cc:654] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a: Neither blocks nor log segments found. Creating new log.
I20260927 06:57:21.468026 13776 tablet_bootstrap.cc:492] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0: Bootstrap starting.
I20260927 06:57:21.469137 13775 log.cc:824] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:21.470260 13777 tablet_bootstrap.cc:654] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1: Neither blocks nor log segments found. Creating new log.
I20260927 06:57:21.470983 13776 tablet_bootstrap.cc:654] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0: Neither blocks nor log segments found. Creating new log.
I20260927 06:57:21.471432 13777 log.cc:824] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:21.471961 13776 log.cc:824] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:21.472438 13775 tablet_bootstrap.cc:492] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a: No bootstrap required, opened a new log
I20260927 06:57:21.472975 13775 ts_tablet_manager.cc:1397] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 06:57:21.474596 13777 tablet_bootstrap.cc:492] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1: No bootstrap required, opened a new log
I20260927 06:57:21.474877 13777 ts_tablet_manager.cc:1397] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 06:57:21.476317 13776 tablet_bootstrap.cc:492] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0: No bootstrap required, opened a new log
I20260927 06:57:21.476626 13776 ts_tablet_manager.cc:1397] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 06:57:21.480525 13775 raft_consensus.cc:348] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221da13bdb3d4d9788589b13e3f7d5e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46283 } } peers { permanent_uuid: "be57c46ae3394c41a516578e130323c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35511 } } peers { permanent_uuid: "d5a14e1135ae418e84b127f1f1cb370a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43189 } }
I20260927 06:57:21.480829 13775 raft_consensus.cc:374] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:21.480955 13775 raft_consensus.cc:729] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5a14e1135ae418e84b127f1f1cb370a, State: Initialized, Role: FOLLOWER
I20260927 06:57:21.481501 13775 consensus_queue.cc:260] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221da13bdb3d4d9788589b13e3f7d5e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46283 } } peers { permanent_uuid: "be57c46ae3394c41a516578e130323c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35511 } } peers { permanent_uuid: "d5a14e1135ae418e84b127f1f1cb370a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43189 } }
I20260927 06:57:21.482515 13747 heartbeater.cc:503] Master 127.0.97.62:46697 was elected leader, sending a full tablet report...
I20260927 06:57:21.484196 13775 ts_tablet_manager.cc:1428] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260927 06:57:21.484676 13777 raft_consensus.cc:348] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221da13bdb3d4d9788589b13e3f7d5e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46283 } } peers { permanent_uuid: "be57c46ae3394c41a516578e130323c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35511 } } peers { permanent_uuid: "d5a14e1135ae418e84b127f1f1cb370a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43189 } }
I20260927 06:57:21.485059 13777 raft_consensus.cc:374] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:21.485174 13777 raft_consensus.cc:729] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221da13bdb3d4d9788589b13e3f7d5e1, State: Initialized, Role: FOLLOWER
I20260927 06:57:21.485694 13777 consensus_queue.cc:260] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221da13bdb3d4d9788589b13e3f7d5e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46283 } } peers { permanent_uuid: "be57c46ae3394c41a516578e130323c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35511 } } peers { permanent_uuid: "d5a14e1135ae418e84b127f1f1cb370a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43189 } }
I20260927 06:57:21.486722 13468 heartbeater.cc:503] Master 127.0.97.62:46697 was elected leader, sending a full tablet report...
I20260927 06:57:21.487306 13777 ts_tablet_manager.cc:1428] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 06:57:21.486428 13776 raft_consensus.cc:348] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221da13bdb3d4d9788589b13e3f7d5e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46283 } } peers { permanent_uuid: "be57c46ae3394c41a516578e130323c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35511 } } peers { permanent_uuid: "d5a14e1135ae418e84b127f1f1cb370a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43189 } }
I20260927 06:57:21.487869 13776 raft_consensus.cc:374] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:21.487960 13776 raft_consensus.cc:729] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be57c46ae3394c41a516578e130323c0, State: Initialized, Role: FOLLOWER
I20260927 06:57:21.488431 13776 consensus_queue.cc:260] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221da13bdb3d4d9788589b13e3f7d5e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46283 } } peers { permanent_uuid: "be57c46ae3394c41a516578e130323c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35511 } } peers { permanent_uuid: "d5a14e1135ae418e84b127f1f1cb370a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43189 } }
I20260927 06:57:21.490063 13776 ts_tablet_manager.cc:1428] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260927 06:57:21.588117 13748 tablet.cc:2369] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:57:21.591890 13614 tablet.cc:2369] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:57:21.701531 13782 raft_consensus.cc:482] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:57:21.701862 13782 raft_consensus.cc:504] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221da13bdb3d4d9788589b13e3f7d5e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46283 } } peers { permanent_uuid: "be57c46ae3394c41a516578e130323c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35511 } } peers { permanent_uuid: "d5a14e1135ae418e84b127f1f1cb370a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43189 } }
I20260927 06:57:21.703673 13782 leader_election.cc:288] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be57c46ae3394c41a516578e130323c0 (127.0.97.2:35511), d5a14e1135ae418e84b127f1f1cb370a (127.0.97.3:43189)
I20260927 06:57:21.712461 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:21.704276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46229 (local address 127.0.97.3:43189)
0927 06:57:21.704487 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:21.704493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:21.705001 (+ 508us) server_negotiation.cc:407] Connection header received
0927 06:57:21.705146 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:21.705150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:21.705229 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:21.705324 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:21.706060 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.707147 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.710076 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.711283 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.711405 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:21.711520 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:21.712097 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 06:57:21.712239 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 06:57:21.712718 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:21.703933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43189 (local address 127.0.97.1:46229)
0927 06:57:21.704893 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:21.704910 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:21.705012 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:21.705345 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:21.705348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:21.705363 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:21.705900 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:21.705909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.707287 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.707290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:21.709902 (+ 2612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:21.709917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.711512 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.711519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:21.711810 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:21.711961 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 06:57:21.712056 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 06:57:21.712237 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":308,"thread_start_us":642,"threads_started":1}
I20260927 06:57:21.712741 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:21.703890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35511 (local address 127.0.97.1:39249)
0927 06:57:21.704646 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:21.704674 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:21.704835 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:21.705312 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:21.705319 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:21.705344 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:21.705874 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:21.705892 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.707285 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.707290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:21.709902 (+ 2612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:21.709917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.711524 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.711529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:21.711811 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:21.711960 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 06:57:21.712056 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 06:57:21.712237 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":344,"thread_start_us":388,"threads_started":1}
I20260927 06:57:21.713088 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:21.704070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39249 (local address 127.0.97.2:35511)
0927 06:57:21.704267 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:21.704274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:21.704786 (+ 512us) server_negotiation.cc:407] Connection header received
0927 06:57:21.705012 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:21.705018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:21.705139 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:21.705235 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:21.706074 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.707147 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.710075 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.711283 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.711405 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:21.711515 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:21.712737 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0927 06:57:21.712857 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 06:57:21.714083 13568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223c30db1d084f1c9b9f4333c3fd2bf1" candidate_uuid: "221da13bdb3d4d9788589b13e3f7d5e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be57c46ae3394c41a516578e130323c0" is_pre_election: true
I20260927 06:57:21.714483 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223c30db1d084f1c9b9f4333c3fd2bf1" candidate_uuid: "221da13bdb3d4d9788589b13e3f7d5e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5a14e1135ae418e84b127f1f1cb370a" is_pre_election: true
I20260927 06:57:21.714604 13568 raft_consensus.cc:2454] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 221da13bdb3d4d9788589b13e3f7d5e1 in term 0.
I20260927 06:57:21.714954 13702 raft_consensus.cc:2454] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 221da13bdb3d4d9788589b13e3f7d5e1 in term 0.
I20260927 06:57:21.715322 13358 leader_election.cc:302] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 221da13bdb3d4d9788589b13e3f7d5e1, be57c46ae3394c41a516578e130323c0; no voters:
I20260927 06:57:21.715754 13782 raft_consensus.cc:2781] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:57:21.715900 13782 raft_consensus.cc:482] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:57:21.715996 13782 raft_consensus.cc:3037] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:21.718346 13782 raft_consensus.cc:504] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221da13bdb3d4d9788589b13e3f7d5e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46283 } } peers { permanent_uuid: "be57c46ae3394c41a516578e130323c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35511 } } peers { permanent_uuid: "d5a14e1135ae418e84b127f1f1cb370a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43189 } }
W20260927 06:57:21.719112 13469 tablet.cc:2369] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:57:21.719487 13568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223c30db1d084f1c9b9f4333c3fd2bf1" candidate_uuid: "221da13bdb3d4d9788589b13e3f7d5e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be57c46ae3394c41a516578e130323c0"
I20260927 06:57:21.719700 13568 raft_consensus.cc:3037] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:21.719844 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223c30db1d084f1c9b9f4333c3fd2bf1" candidate_uuid: "221da13bdb3d4d9788589b13e3f7d5e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5a14e1135ae418e84b127f1f1cb370a"
I20260927 06:57:21.720044 13702 raft_consensus.cc:3037] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:21.722126 13568 raft_consensus.cc:2454] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 221da13bdb3d4d9788589b13e3f7d5e1 in term 1.
I20260927 06:57:21.722575 13782 leader_election.cc:288] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [CANDIDATE]: Term 1 election: Requested vote from peers be57c46ae3394c41a516578e130323c0 (127.0.97.2:35511), d5a14e1135ae418e84b127f1f1cb370a (127.0.97.3:43189)
I20260927 06:57:21.722566 13358 leader_election.cc:302] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 221da13bdb3d4d9788589b13e3f7d5e1, be57c46ae3394c41a516578e130323c0; no voters:
I20260927 06:57:21.722900 13782 raft_consensus.cc:2781] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:57:21.723181 13702 raft_consensus.cc:2454] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 221da13bdb3d4d9788589b13e3f7d5e1 in term 1.
I20260927 06:57:21.723388 13782 raft_consensus.cc:686] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [term 1 LEADER]: Becoming Leader. State: Replica: 221da13bdb3d4d9788589b13e3f7d5e1, State: Running, Role: LEADER
I20260927 06:57:21.724119 13782 consensus_queue.cc:237] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221da13bdb3d4d9788589b13e3f7d5e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46283 } } peers { permanent_uuid: "be57c46ae3394c41a516578e130323c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35511 } } peers { permanent_uuid: "d5a14e1135ae418e84b127f1f1cb370a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43189 } }
I20260927 06:57:21.728681 13270 catalog_manager.cc:5495] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 221da13bdb3d4d9788589b13e3f7d5e1 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "221da13bdb3d4d9788589b13e3f7d5e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221da13bdb3d4d9788589b13e3f7d5e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be57c46ae3394c41a516578e130323c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5a14e1135ae418e84b127f1f1cb370a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43189 } health_report { overall_health: UNKNOWN } } }
I20260927 06:57:21.806944 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:21.801990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45232 (local address 127.0.97.1:46283)
0927 06:57:21.802175 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:21.802179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:21.802195 (+ 16us) server_negotiation.cc:407] Connection header received
0927 06:57:21.802335 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:21.802339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:21.802424 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:21.802543 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:57:21.802866 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.803577 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.804913 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.805320 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.805431 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:21.805829 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:57:21.806396 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:57:21.806534 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:21.806572 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 06:57:21.806634 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 06:57:21.814862 13702 raft_consensus.cc:1261] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a [term 1 FOLLOWER]: Refusing update from remote peer 221da13bdb3d4d9788589b13e3f7d5e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:57:21.814860 13568 raft_consensus.cc:1261] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0 [term 1 FOLLOWER]: Refusing update from remote peer 221da13bdb3d4d9788589b13e3f7d5e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:57:21.815971 13782 consensus_queue.cc:1035] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5a14e1135ae418e84b127f1f1cb370a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:57:21.816342 13786 consensus_queue.cc:1035] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be57c46ae3394c41a516578e130323c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:57:21.825793 13790 mvcc.cc:205] Tried to move back new op lower bound from 7333856222463164416 to 7333856222106161152. Current Snapshot: MvccSnapshot[applied={T|T < 7333856222463164416}]
I20260927 06:57:21.844607 13270 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:57:21.844864 13270 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:57:21.847918 13270 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6097204e6b7b4fe49b5ebc4aac813fa0: Sending DeleteTablet for 3 replicas of tablet 223c30db1d084f1c9b9f4333c3fd2bf1
I20260927 06:57:21.848943 13548 tablet_service.cc:1514] Processing DeleteTablet for tablet 223c30db1d084f1c9b9f4333c3fd2bf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:57:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44074
I20260927 06:57:21.849069 13682 tablet_service.cc:1514] Processing DeleteTablet for tablet 223c30db1d084f1c9b9f4333c3fd2bf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:57:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45612
I20260927 06:57:21.849078 13403 tablet_service.cc:1514] Processing DeleteTablet for tablet 223c30db1d084f1c9b9f4333c3fd2bf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:57:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45228
I20260927 06:57:21.855190 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 13308
I20260927 06:57:21.864184 13799 tablet_replica.cc:330] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0: stopping tablet replica
I20260927 06:57:21.864754 13799 raft_consensus.cc:2229] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:57:21.865365 13799 raft_consensus.cc:2258] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:57:21.867835 13800 tablet_replica.cc:330] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1: stopping tablet replica
I20260927 06:57:21.868311 13800 raft_consensus.cc:2229] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:57:21.869081 13800 raft_consensus.cc:2258] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:57:21.871279 13800 ts_tablet_manager.cc:1905] T 223c30db1d084f1c9b9f4333c3fd2bf1 P 221da13bdb3d4d9788589b13e3f7d5e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:57:21.871269 13799 ts_tablet_manager.cc:1905] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:57:21.872161 13801 tablet_replica.cc:330] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a: stopping tablet replica
I20260927 06:57:21.872524 13801 raft_consensus.cc:2229] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:57:21.872991 13801 raft_consensus.cc:2258] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:57:21.877986 13801 ts_tablet_manager.cc:1905] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:57:21.878509 13799 ts_tablet_manager.cc:1918] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:57:21.878656 13799 log.cc:1196] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/223c30db1d084f1c9b9f4333c3fd2bf1
I20260927 06:57:21.879182 13799 ts_tablet_manager.cc:1939] T 223c30db1d084f1c9b9f4333c3fd2bf1 P be57c46ae3394c41a516578e130323c0: Deleting consensus metadata
I20260927 06:57:21.881094 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:21.857310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.0.97.1:46283)
0927 06:57:21.857470 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:21.857475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:21.857491 (+ 16us) server_negotiation.cc:407] Connection header received
0927 06:57:21.857542 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:21.857547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:21.857628 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:21.857720 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:21.860412 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.861634 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.863864 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:21.864558 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:21.864677 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:21.870841 (+ 6164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:21.870854 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:21.870870 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:21.870916 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:21.875216 (+ 4300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:21.876053 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:21.876066 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:21.876069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:21.876229 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:21.876679 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:21.876684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:21.876688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:21.877123 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:21.880244 (+ 3121us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:21.880389 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 06:57:21.880628 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 06:57:21.881022 13257 catalog_manager.cc:4841] TS be57c46ae3394c41a516578e130323c0 (127.0.97.2:35511): tablet 223c30db1d084f1c9b9f4333c3fd2bf1 (table test-table [id=3ee2b89f1897442f849a563885934aa6]) successfully deleted
I20260927 06:57:21.882169 13463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:21.884363 13801 ts_tablet_manager.cc:1918] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:57:21.884502 13801 log.cc:1196] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/223c30db1d084f1c9b9f4333c3fd2bf1
I20260927 06:57:21.884984 13801 ts_tablet_manager.cc:1939] T 223c30db1d084f1c9b9f4333c3fd2bf1 P d5a14e1135ae418e84b127f1f1cb370a: Deleting consensus metadata
I20260927 06:57:21.886433 13255 catalog_manager.cc:4841] TS d5a14e1135ae418e84b127f1f1cb370a (127.0.97.3:43189): tablet 223c30db1d084f1c9b9f4333c3fd2bf1 (table test-table [id=3ee2b89f1897442f849a563885934aa6]) successfully deleted
I20260927 06:57:21.987428 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 13308
W20260927 06:57:22.005188 13258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:46283 (error 108)
I20260927 06:57:22.005322 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 13472
I20260927 06:57:22.007108 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:22.006361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46283 (local address 127.0.0.1:45238)
0927 06:57:22.006789 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:22.006879 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":237,"threads_started":1}
W20260927 06:57:22.007369 13258 catalog_manager.cc:4584] TS 221da13bdb3d4d9788589b13e3f7d5e1 (127.0.97.1:46283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 223c30db1d084f1c9b9f4333c3fd2bf1: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46283: connect: Connection refused (error 111)
I20260927 06:57:22.022826 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:22.006070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44086 (local address 127.0.97.2:35511)
0927 06:57:22.006243 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:22.006250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:22.006273 (+ 23us) server_negotiation.cc:407] Connection header received
0927 06:57:22.006479 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:22.006491 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:22.006560 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:22.006671 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:22.008172 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:22.008850 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:22.010128 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:22.010569 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:22.010668 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:22.017453 (+ 6785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:22.017466 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:22.017480 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:22.017512 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:22.020173 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:22.020795 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:22.020801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:22.020803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:22.020928 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:22.021298 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:22.021301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:22.021303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:22.021707 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:22.021945 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:22.022269 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 06:57:22.022423 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 06:57:22.023428 13608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:22.060961 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:22.060513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46283 (local address 127.0.0.1:45248)
0927 06:57:22.060721 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:22.060822 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 06:57:22.061976 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:22.061650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46283 (local address 127.0.0.1:45256)
0927 06:57:22.061791 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:22.061856 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 06:57:22.110508 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 13472
I20260927 06:57:22.122773 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:22.122270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46283 (local address 127.0.0.1:45266)
0927 06:57:22.122476 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:22.122631 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 06:57:22.123808 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:22.123460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46283 (local address 127.0.0.1:45270)
0927 06:57:22.123599 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:22.123673 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 06:57:22.127206 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 13617
I20260927 06:57:22.139562 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:22.127854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45616 (local address 127.0.97.3:43189)
0927 06:57:22.128007 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:22.128011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:22.128028 (+ 17us) server_negotiation.cc:407] Connection header received
0927 06:57:22.128116 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:22.128119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:22.128180 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:22.128268 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:22.129384 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:22.130121 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:22.131241 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:22.131625 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:22.131709 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:22.133957 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:22.133969 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:22.133983 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:22.134194 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:22.137171 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:22.137750 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:22.137756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:22.137758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:22.137874 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:22.138264 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:22.138267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:22.138269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:22.138572 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:22.138826 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:22.139153 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 06:57:22.139318 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 06:57:22.140158 13742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:22.204437 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:22.204004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46283 (local address 127.0.0.1:45280)
0927 06:57:22.204198 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:22.204285 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 06:57:22.205464 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:22.205102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46283 (local address 127.0.0.1:45296)
0927 06:57:22.205278 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:22.205352 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 06:57:22.225793 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 13617
I20260927 06:57:22.241995 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 13238
I20260927 06:57:22.243157 13301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:22.352432 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 13238
I20260927 06:57:22.379437 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 06:57:22.714115 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 06:57:22.714360 388 mini_ranger.cc:66] Stopping Ranger...
I20260927 06:57:23.059389 388 mini_ranger.cc:68] Stopped Ranger
2026-09-27 06:57:23.059 UTC [12290] LOG: received smart shutdown request
2026-09-27 06:57:23.063 UTC [12290] LOG: background worker "logical replication launcher" (PID 12299) exited with exit code 1
2026-09-27 06:57:23.063 UTC [12294] LOG: shutting down
2026-09-27 06:57:23.215 UTC [12290] LOG: database system is shut down
2026-09-27T06:57:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51213 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:57:23 dist-test-slave-f892 krb5kdc[13816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:57:23 dist-test-slave-f892 krb5kdc[13816](info): set up 2 sockets
Sep 27 06:57:23 dist-test-slave-f892 krb5kdc[13816](info): commencing operation
krb5kdc: starting...
W20260927 06:57:24.426419 388 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:57:24 dist-test-slave-f892 krb5kdc[13816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:57:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:57:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:57:24.649566 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36295
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36295,127.0.97.61:40221,127.0.97.60:33977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:57:24.968531 13829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:24.968799 13829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:24.968924 13829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:24.975908 13829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:57:24.976027 13829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:24.976112 13829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:24.976212 13829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:57:24.976289 13829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:57:24.985625 13829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36295,127.0.97.61:40221,127.0.97.60:33977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:24.986873 13829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:24.988242 13829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:24.999666 13834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:24.999666 13837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:24.999745 13835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:25.000772 13829 server_base.cc:1034] running on GCE node
I20260927 06:57:25.001485 13829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:25.002838 13829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:25.004066 13829 hybrid_clock.cc:648] HybridClock initialized: now 1790492245004030 us; error 53 us; skew 500 ppm
I20260927 06:57:25.004400 13829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:25 dist-test-slave-f892 krb5kdc[13816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:25.007452 13829 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:57:25.008787 13829 webserver.cc:460] Webserver started at http://127.0.97.62:42693/ using document root <none> and password file <none>
I20260927 06:57:25.009440 13829 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:25.009584 13829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:25.009874 13829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:57:25.012742 13829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "747819f0ab144ec1bb521fe514e08595"
format_stamp: "Formatted at 2026-09-27 06:57:25 on dist-test-slave-f892"
server_key: "11160ad368084191477f3ca282196b32"
server_key_iv: "c0677f0a0b4b00c9c811a958dac889e1"
server_key_version: "encryptionkey@0"
I20260927 06:57:25.013507 13829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "747819f0ab144ec1bb521fe514e08595"
format_stamp: "Formatted at 2026-09-27 06:57:25 on dist-test-slave-f892"
server_key: "11160ad368084191477f3ca282196b32"
server_key_iv: "c0677f0a0b4b00c9c811a958dac889e1"
server_key_version: "encryptionkey@0"
I20260927 06:57:25.018530 13829 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 06:57:25.021852 13844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:25.023217 13829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 06:57:25.023401 13829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "747819f0ab144ec1bb521fe514e08595"
format_stamp: "Formatted at 2026-09-27 06:57:25 on dist-test-slave-f892"
server_key: "11160ad368084191477f3ca282196b32"
server_key_iv: "c0677f0a0b4b00c9c811a958dac889e1"
server_key_version: "encryptionkey@0"
I20260927 06:57:25.023604 13829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:25.041533 13829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:25.045873 13829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:25.046231 13829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:25.064466 13829 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36295
I20260927 06:57:25.064502 13895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36295 every 8 connection(s)
I20260927 06:57:25.065769 13829 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:57:25.069437 13896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:25.069530 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 13829
I20260927 06:57:25.069772 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:57:25.070606 388 external_mini_cluster.cc:1428] Setting key 3b3c20f942226bbb6d551688a8334118
I20260927 06:57:25.079352 13896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 27 06:57:25 dist-test-slave-f892 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:25.104722 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.082339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60024 (local address 127.0.97.62:36295)
0927 06:57:25.083975 (+ 1636us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:25.083984 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:25.084028 (+ 44us) server_negotiation.cc:407] Connection header received
0927 06:57:25.085505 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:25.085511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:25.085790 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:25.086042 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:25.087688 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:25.088949 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:25.091266 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:25.091821 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:25.091956 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:25.096637 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:25.096650 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:25.096668 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:25.096716 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:25.100626 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:25.101289 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:25.101301 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:25.101307 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:25.101507 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:25.101873 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:25.101880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:25.101886 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:25.102609 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:25.102818 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:25.103464 (+ 646us) server_negotiation.cc:299] Negotiation successful
0927 06:57:25.103727 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1437,"thread_start_us":1230,"threads_started":1}
I20260927 06:57:25.104810 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.082330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36295 (local address 127.0.0.1:60024)
0927 06:57:25.083200 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.083243 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:25.085158 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:25.086117 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:25.086127 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:25.086960 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:25.087475 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:25.087504 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:25.089117 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:25.089126 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:25.091111 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:25.091128 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:25.092083 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:25.092093 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:25.092306 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:25.093003 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:25.093025 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:25.096356 (+ 3331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:25.100751 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:25.100760 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:25.100768 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:25.101138 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:25.101614 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:25.101617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:25.101619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:25.101757 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:25.102756 (+ 999us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:25.102774 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:25.103049 (+ 275us) client_negotiation.cc:769] Sending connection context
0927 06:57:25.103332 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 06:57:25.103948 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":71,"thread_start_us":408,"threads_started":1}
I20260927 06:57:25.107518 13896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } has no permanent_uuid. Determining permanent_uuid...
I20260927 06:57:25.108639 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.108271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40221 (local address 127.0.0.1:56122)
0927 06:57:25.108431 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.108540 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 06:57:25.108996 13845 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40221: connect: Connection refused (error 111)
I20260927 06:57:25.110491 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.110175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40221 (local address 127.0.0.1:56136)
0927 06:57:25.110303 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.110368 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 06:57:25.110780 13896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40221: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40221: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 06:57:25.125620 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40221
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36295,127.0.97.61:40221,127.0.97.60:33977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:57:25.160106 13896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } attempt: 1
I20260927 06:57:25.161301 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.160847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40221 (local address 127.0.0.1:56142)
0927 06:57:25.161058 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.161148 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 06:57:25.162422 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.162079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40221 (local address 127.0.0.1:56152)
0927 06:57:25.162211 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.162277 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 06:57:25.162686 13896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40221: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40221: connect: Connection refused (error 111)
I20260927 06:57:25.230917 13896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } attempt: 2
I20260927 06:57:25.232113 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.231690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40221 (local address 127.0.0.1:56164)
0927 06:57:25.231887 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.231973 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 06:57:25.233171 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.232827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40221 (local address 127.0.0.1:56180)
0927 06:57:25.232955 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.233032 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 06:57:25.233424 13896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40221: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40221: connect: Connection refused (error 111)
I20260927 06:57:25.324771 13896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } attempt: 3
I20260927 06:57:25.326109 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.325592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40221 (local address 127.0.0.1:56192)
0927 06:57:25.325798 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.325885 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 06:57:25.327195 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.326833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40221 (local address 127.0.0.1:56194)
0927 06:57:25.326989 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.327073 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 06:57:25.327458 13896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40221: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40221: connect: Connection refused (error 111)
W20260927 06:57:25.444591 13903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:25.444881 13903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:25.445001 13903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:25.452241 13903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:57:25.452347 13903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:25.452440 13903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:25.452535 13903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:57:25.452625 13903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:57:25.462299 13903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36295,127.0.97.61:40221,127.0.97.60:33977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:25.463622 13903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:25.464870 13903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 06:57:25.470693 13896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } attempt: 4
I20260927 06:57:25.471849 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.471339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40221 (local address 127.0.0.1:56196)
0927 06:57:25.471633 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.471727 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260927 06:57:25.472820 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.472452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40221 (local address 127.0.0.1:56202)
0927 06:57:25.472615 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.472700 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 06:57:25.473023 13896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40221: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40221: connect: Connection refused (error 111)
W20260927 06:57:25.477047 13908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:25.477097 13909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:25.477129 13911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:25.478267 13903 server_base.cc:1034] running on GCE node
I20260927 06:57:25.479044 13903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:25.480311 13903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:25.481526 13903 hybrid_clock.cc:648] HybridClock initialized: now 1790492245481454 us; error 40 us; skew 500 ppm
I20260927 06:57:25.481840 13903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:25 dist-test-slave-f892 krb5kdc[13816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:25.484907 13903 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 06:57:25.486442 13903 webserver.cc:460] Webserver started at http://127.0.97.61:34667/ using document root <none> and password file <none>
I20260927 06:57:25.487216 13903 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:25.487370 13903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:25.487677 13903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:57:25.490535 13903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-1/data/instance:
uuid: "ddd6ad857f9d4f888a12c54ad3cc9246"
format_stamp: "Formatted at 2026-09-27 06:57:25 on dist-test-slave-f892"
server_key: "11372b79cc0703b4410181f20a2a6126"
server_key_iv: "88a6347be123d0562ecc81bc9edc9fec"
server_key_version: "encryptionkey@0"
I20260927 06:57:25.491283 13903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-1/wal/instance:
uuid: "ddd6ad857f9d4f888a12c54ad3cc9246"
format_stamp: "Formatted at 2026-09-27 06:57:25 on dist-test-slave-f892"
server_key: "11372b79cc0703b4410181f20a2a6126"
server_key_iv: "88a6347be123d0562ecc81bc9edc9fec"
server_key_version: "encryptionkey@0"
I20260927 06:57:25.496069 13903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 06:57:25.499197 13918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:25.500376 13903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 06:57:25.500566 13903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-1/wal
uuid: "ddd6ad857f9d4f888a12c54ad3cc9246"
format_stamp: "Formatted at 2026-09-27 06:57:25 on dist-test-slave-f892"
server_key: "11372b79cc0703b4410181f20a2a6126"
server_key_iv: "88a6347be123d0562ecc81bc9edc9fec"
server_key_version: "encryptionkey@0"
I20260927 06:57:25.500759 13903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:25.524187 13903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:25.528416 13903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:25.528692 13903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:25.545581 13903 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40221
I20260927 06:57:25.545622 13969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40221 every 8 connection(s)
I20260927 06:57:25.546909 13903 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-1/data/info.pb
I20260927 06:57:25.550606 13970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:25.555402 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 13903
I20260927 06:57:25.555622 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-1/wal/instance
I20260927 06:57:25.556104 388 external_mini_cluster.cc:1428] Setting key 3b1d0153e62d299e6b2babd820004b0c
I20260927 06:57:25.557708 13970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 06:57:25 dist-test-slave-f892 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260927 06:57:25.579380 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.561021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60040 (local address 127.0.97.62:36295)
0927 06:57:25.561268 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:25.561277 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:25.562545 (+ 1268us) server_negotiation.cc:407] Connection header received
0927 06:57:25.564535 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:25.564543 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:25.564636 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:25.564753 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:25.566362 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:25.567532 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:25.569065 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:25.569818 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:25.569959 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:25.573686 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:25.573691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:25.573694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:25.573726 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:25.576386 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:25.577129 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:25.577135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:25.577140 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:25.577260 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:25.577670 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:25.577675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:25.577678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:25.577941 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:25.578119 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:25.578819 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 06:57:25.579039 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 06:57:25.580322 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.561477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36295 (local address 127.0.0.1:60040)
0927 06:57:25.562327 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.562389 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:25.564087 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:25.564996 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:25.565004 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:25.565538 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:25.566170 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:25.566190 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:25.567692 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:25.567696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:25.568912 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:25.568923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:25.570122 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:25.570130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:25.570312 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:25.570991 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:25.571017 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:25.573470 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:25.576552 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:25.576560 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:25.576576 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:25.576993 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:25.577371 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:25.577376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:25.577380 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:25.577550 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:25.578111 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:25.578119 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:25.578472 (+ 353us) client_negotiation.cc:769] Sending connection context
0927 06:57:25.578729 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 06:57:25.579223 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":387,"threads_started":1}
I20260927 06:57:25.582212 13970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 06:57:25 dist-test-slave-f892 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 06:57:25.598425 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.583016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40221 (local address 127.0.0.1:56214)
0927 06:57:25.583196 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.583218 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:25.583343 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:25.584214 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:25.584219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:25.584560 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:25.584687 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:25.584695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:25.586478 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:25.586484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:25.587519 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:25.587531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:25.588781 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:25.588786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:25.588939 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:25.589543 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:25.589564 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:25.591761 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:25.596026 (+ 4265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:25.596032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:25.596035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:25.596404 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:25.596815 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:25.596818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:25.596820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:25.596953 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:25.597700 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:25.597703 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:25.597822 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 06:57:25.597924 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 06:57:25.598178 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 06:57:25.598850 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.583112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56214 (local address 127.0.97.61:40221)
0927 06:57:25.583731 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:25.583741 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:25.583772 (+ 31us) server_negotiation.cc:407] Connection header received
0927 06:57:25.583826 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:25.583835 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:25.584055 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:25.584204 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:25.584836 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:25.586322 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:25.587659 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:25.588600 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:25.588747 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:25.591972 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:25.591988 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:25.592006 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:25.592052 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:25.595874 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:25.596537 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:25.596548 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:25.596554 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:25.596701 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:25.597060 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:25.597067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:25.597081 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:25.597570 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:25.597801 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:25.598137 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 06:57:25.598401 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":285,"threads_started":1}
I20260927 06:57:25.601055 13970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } has no permanent_uuid. Determining permanent_uuid...
I20260927 06:57:25.602185 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.601701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48186)
0927 06:57:25.601871 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.601978 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 06:57:25.602451 13920 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
I20260927 06:57:25.603969 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.603656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48196)
0927 06:57:25.603794 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.603877 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 06:57:25.604283 13970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33977: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 06:57:25.607316 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33977
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36295,127.0.97.61:40221,127.0.97.60:33977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:57:25.653604 13970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } attempt: 1
I20260927 06:57:25.654886 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.654423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48202)
0927 06:57:25.654627 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.654734 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 06:57:25.655910 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.655581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48208)
0927 06:57:25.655745 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.655809 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 06:57:25.656129 13970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33977: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
I20260927 06:57:25.724395 13970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } attempt: 2
I20260927 06:57:25.725517 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.725068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48216)
0927 06:57:25.725269 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.725359 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 06:57:25.726620 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.726192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48226)
0927 06:57:25.726364 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.726456 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 06:57:25.726882 13970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33977: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
I20260927 06:57:25.772368 13896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } attempt: 5
Sep 27 06:57:25 dist-test-slave-f892 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 06:57:25.786670 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.773166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40221 (local address 127.0.0.1:56224)
0927 06:57:25.773413 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.773437 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:25.773577 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:25.773926 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:25.773931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:25.774464 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:25.774693 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:25.774705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:25.775663 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:25.775669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:25.777028 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:25.777053 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:25.777777 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:25.777783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:25.777979 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:25.778625 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:25.778651 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:25.781226 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:25.784164 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:25.784169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:25.784174 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:25.784526 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:25.784886 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:25.784891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:25.784895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:25.785029 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:25.785673 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:25.785678 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:25.785843 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 06:57:25.785989 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 06:57:25.786284 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 06:57:25.786671 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.773420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.0.97.61:40221)
0927 06:57:25.773591 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:25.773595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:25.773614 (+ 19us) server_negotiation.cc:407] Connection header received
0927 06:57:25.773697 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:25.773701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:25.773772 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:25.773855 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:25.774850 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:25.775519 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:25.777203 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:25.777630 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:25.777726 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:25.781419 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:25.781424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:25.781427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:25.781456 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:25.784028 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:25.784654 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:25.784659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:25.784667 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:25.784778 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:25.785139 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:25.785146 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:25.785149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:25.785441 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:25.785711 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:25.786132 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 06:57:25.786354 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 06:57:25.787685 13896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } has no permanent_uuid. Determining permanent_uuid...
I20260927 06:57:25.788774 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.788381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48230)
0927 06:57:25.788559 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.788641 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 06:57:25.789809 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.789489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48240)
0927 06:57:25.789624 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.789712 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 06:57:25.790143 13896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33977: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
I20260927 06:57:25.818114 13970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } attempt: 3
I20260927 06:57:25.819386 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.818730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48250)
0927 06:57:25.819006 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.819135 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 06:57:25.820536 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.820069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48262)
0927 06:57:25.820298 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.820392 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260927 06:57:25.820775 13970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33977: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
I20260927 06:57:25.841463 13896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } attempt: 1
I20260927 06:57:25.842463 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.842074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48270)
0927 06:57:25.842243 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.842319 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 06:57:25.843374 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.843054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48274)
0927 06:57:25.843197 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.843272 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 06:57:25.843568 13896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33977: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
I20260927 06:57:25.911789 13896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } attempt: 2
I20260927 06:57:25.912868 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.912407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48282)
0927 06:57:25.912620 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.912711 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 06:57:25.913769 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.913452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48298)
0927 06:57:25.913594 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.913668 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 06:57:25.913972 13896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33977: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
W20260927 06:57:25.926514 13977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:25.926836 13977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:25.926975 13977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:25.934182 13977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:57:25.934306 13977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:25.934405 13977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:25.934506 13977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:57:25.934582 13977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:57:25.944053 13977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36295,127.0.97.61:40221,127.0.97.60:33977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:25.945281 13977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:25.946512 13977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:25.958164 13982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:25.958226 13985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:25.958316 13983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:25.959040 13977 server_base.cc:1034] running on GCE node
I20260927 06:57:25.959707 13977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:25.960856 13977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:25.962105 13977 hybrid_clock.cc:648] HybridClock initialized: now 1790492245962061 us; error 55 us; skew 500 ppm
I20260927 06:57:25.962430 13977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:57:25.963979 13970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } attempt: 4
I20260927 06:57:25.965003 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.964533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48304)
0927 06:57:25.964773 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.964859 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 06:57:25.965824 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:25.965563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48312)
0927 06:57:25.965675 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:25.965731 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 06:57:25.966096 13970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33977: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Sep 27 06:57:25 dist-test-slave-f892 krb5kdc[13816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:25.968406 13977 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 06:57:25.969704 13977 webserver.cc:460] Webserver started at http://127.0.97.60:35165/ using document root <none> and password file <none>
I20260927 06:57:25.970368 13977 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:25.970464 13977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:25.970707 13977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:57:25.973933 13977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-2/data/instance:
uuid: "f8459be6adaf449386b9adb1a3a3d005"
format_stamp: "Formatted at 2026-09-27 06:57:25 on dist-test-slave-f892"
server_key: "006ee8bc852fe6d1727d8665ce4581b4"
server_key_iv: "e8e01505d765cf1c94f1e3ca64da9eae"
server_key_version: "encryptionkey@0"
I20260927 06:57:25.974743 13977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-2/wal/instance:
uuid: "f8459be6adaf449386b9adb1a3a3d005"
format_stamp: "Formatted at 2026-09-27 06:57:25 on dist-test-slave-f892"
server_key: "006ee8bc852fe6d1727d8665ce4581b4"
server_key_iv: "e8e01505d765cf1c94f1e3ca64da9eae"
server_key_version: "encryptionkey@0"
I20260927 06:57:25.979398 13977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 06:57:25.982427 13992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:25.983632 13977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 06:57:25.983773 13977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-2/wal
uuid: "f8459be6adaf449386b9adb1a3a3d005"
format_stamp: "Formatted at 2026-09-27 06:57:25 on dist-test-slave-f892"
server_key: "006ee8bc852fe6d1727d8665ce4581b4"
server_key_iv: "e8e01505d765cf1c94f1e3ca64da9eae"
server_key_version: "encryptionkey@0"
I20260927 06:57:25.983933 13977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:26.014976 13977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:26.018894 13977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:26.019115 13977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:26.020203 13896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } attempt: 3
I20260927 06:57:26.021252 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:26.020829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48320)
0927 06:57:26.021029 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:26.021106 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 06:57:26.022279 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:26.021870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48332)
0927 06:57:26.022100 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:26.022170 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260927 06:57:26.022477 13896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33977: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33977: connect: Connection refused (error 111)
I20260927 06:57:26.037901 13977 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33977
I20260927 06:57:26.037936 14043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33977 every 8 connection(s)
I20260927 06:57:26.039131 13977 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-2/data/info.pb
I20260927 06:57:26.042496 14044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:26.046882 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 13977
I20260927 06:57:26.047072 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492047466241-388-0/minicluster-data/master-2/wal/instance
I20260927 06:57:26.047577 388 external_mini_cluster.cc:1428] Setting key 2a44c296af05ccfb5857ac4fe46fab9e
I20260927 06:57:26.049335 14044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 06:57:26 dist-test-slave-f892 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 27 06:57:26 dist-test-slave-f892 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 06:57:26.069905 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:26.051950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.0.97.62:36295)
0927 06:57:26.052115 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:26.052119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:26.052814 (+ 695us) server_negotiation.cc:407] Connection header received
0927 06:57:26.053101 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:26.053107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:26.053185 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:26.053298 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:26.054876 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.055876 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.057900 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.058579 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.058721 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:26.062530 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:26.062535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:26.062538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:26.062568 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:26.065817 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.066622 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.066628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.066632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.066802 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.067356 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.067362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.067365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.067672 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:26.067814 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:26.069340 (+ 1526us) server_negotiation.cc:299] Negotiation successful
0927 06:57:26.069552 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 06:57:26.070199 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:26.049204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.0.97.60:33977)
0927 06:57:26.049918 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:26.049929 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:26.049970 (+ 41us) server_negotiation.cc:407] Connection header received
0927 06:57:26.051880 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:26.051914 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:26.052417 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:26.052796 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:26.054495 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.056176 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.057967 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.058791 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.058970 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:26.063310 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:26.063317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:26.063327 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:26.063354 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:26.066280 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.067138 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.067145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.067148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.067285 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.067685 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.067695 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.067697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.068193 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:26.068470 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:26.068818 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 06:57:26.069007 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":535,"spinlock_wait_cycles":17152,"thread_start_us":323,"threads_started":1}
I20260927 06:57:26.070564 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:26.051981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36295 (local address 127.0.0.1:60054)
0927 06:57:26.052646 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:26.052710 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:26.052946 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:26.053375 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:26.053383 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:26.054179 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:26.054697 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:26.054712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.056039 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.056042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:26.057768 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:26.057782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.058778 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.058783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:26.058977 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:26.059678 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:26.059708 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:26.062369 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:26.065950 (+ 3581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:26.065960 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:26.065971 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:26.066515 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:26.066940 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:26.066944 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:26.066948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:26.067222 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:26.067794 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:26.067800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:26.068184 (+ 384us) client_negotiation.cc:769] Sending connection context
0927 06:57:26.068452 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 06:57:26.068888 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":29,"thread_start_us":363,"threads_started":1}
I20260927 06:57:26.072181 14044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 06:57:26 dist-test-slave-f892 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 27 06:57:26 dist-test-slave-f892 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:26.084903 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:26.072925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56238 (local address 127.0.97.61:40221)
0927 06:57:26.073112 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:26.073118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:26.073136 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:57:26.073203 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:26.073209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:26.073277 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:26.073374 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:26.074067 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.074972 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.076162 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.076568 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.076660 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:26.079562 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:26.079567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:26.079570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:26.079599 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:26.082268 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.082907 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.082911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.082913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.083038 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.083480 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.083487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.083491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.083824 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:26.083941 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:26.084348 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 06:57:26.084609 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 06:57:26.084945 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:26.072819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40221 (local address 127.0.0.1:56238)
0927 06:57:26.072966 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:26.072982 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:26.073066 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:26.073407 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:26.073410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:26.073694 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:26.073818 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:26.073825 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.075126 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.075129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:26.076009 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:26.076023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.076703 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.076710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:26.076927 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:26.077475 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:26.077490 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:26.079383 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:26.082399 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:26.082404 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:26.082408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:26.082762 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:26.083146 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:26.083151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:26.083154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:26.083339 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:26.083943 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:26.083951 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:26.084109 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 06:57:26.084266 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 06:57:26.084534 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 06:57:26.086126 14044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } has no permanent_uuid. Determining permanent_uuid...
I20260927 06:57:26.089375 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:26.074197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.0.97.62:36295)
0927 06:57:26.074350 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:26.074355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:26.074371 (+ 16us) server_negotiation.cc:407] Connection header received
0927 06:57:26.074414 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:26.074419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:26.074481 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:26.074581 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:26.075943 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.076796 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.078287 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.078743 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.078834 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:26.082897 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:26.082910 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:26.082914 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:26.082950 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:26.086089 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.086830 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.086835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.086839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.087004 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.087455 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.087460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.087464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.087723 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:26.088006 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:26.088870 (+ 864us) server_negotiation.cc:299] Negotiation successful
0927 06:57:26.089062 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 06:57:26 dist-test-slave-f892 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 06:57:26 dist-test-slave-f892 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 06:57:26.098867 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:26.086910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48342)
0927 06:57:26.087085 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:26.087101 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:26.087228 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:26.087516 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:26.087520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:26.087860 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:26.088017 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:26.088028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.089295 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.089298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:26.090425 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:26.090436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.091251 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.091257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:26.091428 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:26.091944 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:26.091959 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:26.093696 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:26.096558 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:26.096563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:26.096567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:26.096879 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:26.097229 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:26.097233 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:26.097236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:26.097353 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:26.097813 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:26.097817 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:26.098062 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 06:57:26.098231 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 06:57:26.098505 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 06:57:26.098970 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:26.086854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.0.97.60:33977)
0927 06:57:26.087024 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:26.087030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:26.087182 (+ 152us) server_negotiation.cc:407] Connection header received
0927 06:57:26.087337 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:26.087342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:26.087399 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:26.087503 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:26.088213 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.089178 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.090549 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.091138 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.091238 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:26.093858 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:26.093864 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:26.093868 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:26.093910 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:26.096421 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.096984 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.096992 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.096996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.097115 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.097463 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.097467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.097469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.097699 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:26.097801 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:26.098310 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 06:57:26.098641 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":57}
I20260927 06:57:26.104547 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:26.090850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56248 (local address 127.0.97.61:40221)
0927 06:57:26.090982 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:26.090985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:26.090999 (+ 14us) server_negotiation.cc:407] Connection header received
0927 06:57:26.091138 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:26.091142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:26.091188 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:26.091290 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:26.092523 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.093202 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.094476 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.094853 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.094983 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:26.098780 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:26.098785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:26.098789 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:26.098827 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:26.102084 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.102716 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.102723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.102728 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.102859 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.103186 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.103190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.103193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.103550 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:26.103832 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:26.103956 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 06:57:26.104233 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 06:57:26.104791 14044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005: Bootstrap starting.
I20260927 06:57:26.108796 14044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005: Neither blocks nor log segments found. Creating new log.
I20260927 06:57:26.109231 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:57:26.110042 14044 log.cc:824] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:26.113112 14044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005: No bootstrap required, opened a new log
I20260927 06:57:26.127995 14044 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } }
I20260927 06:57:26.128507 14044 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:26.128680 14044 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8459be6adaf449386b9adb1a3a3d005, State: Initialized, Role: FOLLOWER
I20260927 06:57:26.129406 14044 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } }
I20260927 06:57:26.137769 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:26.114747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.0.97.62:36295)
0927 06:57:26.114930 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:26.114938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:26.115767 (+ 829us) server_negotiation.cc:407] Connection header received
0927 06:57:26.116034 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:26.116040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:26.116123 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:26.116199 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:26.119221 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.120680 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.123695 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.124521 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.124645 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:26.128434 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:26.128441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:26.128445 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:26.128486 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:26.133084 (+ 4598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.134711 (+ 1627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.134719 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.134726 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.134930 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.135542 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.135550 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.135555 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.135995 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:26.136285 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:26.136948 (+ 663us) server_negotiation.cc:299] Negotiation successful
0927 06:57:26.137251 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 06:57:26.141240 14057 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } } }
I20260927 06:57:26.141703 14057 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:57:26.142627 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:26.116148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.0.97.60:33977)
0927 06:57:26.116341 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:26.116349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:26.124984 (+ 8635us) server_negotiation.cc:407] Connection header received
0927 06:57:26.125223 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:26.125228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:26.125308 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:26.125376 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:26.127806 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.128835 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.130969 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.131752 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.131832 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:26.135572 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:26.135578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:26.135582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:26.135614 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:26.139220 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.140089 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.140096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.140102 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.140280 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.140747 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.140752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.140755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.141199 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:26.141436 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:26.142055 (+ 619us) server_negotiation.cc:299] Negotiation successful
0927 06:57:26.142393 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 06:57:26.142719 14044 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:57:26.142807 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:26.115231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56252 (local address 127.0.97.61:40221)
0927 06:57:26.115693 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:26.115700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:26.124896 (+ 9196us) server_negotiation.cc:407] Connection header received
0927 06:57:26.124966 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:26.124972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:26.125048 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:26.125140 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:26.127517 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.128619 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.130971 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.131656 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.131784 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:26.135567 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:26.135573 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:26.135577 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:26.135615 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:26.139455 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.140315 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.140323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.140327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.140469 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.140884 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.140891 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.140894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.141216 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:26.141436 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:26.142230 (+ 794us) server_negotiation.cc:299] Negotiation successful
0927 06:57:26.142453 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347}
W20260927 06:57:26.152150 14068 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 06:57:26.152325 14068 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 06:57:26.148973 13977 master_runner.cc:186] Error getting master registration for 127.0.97.62:36295: OK, instance_id { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" instance_seqno: 1790492245059234 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 06:57:26.157876 13977 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 06:57:26.196944 14057 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:57:26.197264 14057 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } }
I20260927 06:57:26.199064 14057 leader_election.cc:288] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 747819f0ab144ec1bb521fe514e08595 (127.0.97.62:36295), ddd6ad857f9d4f888a12c54ad3cc9246 (127.0.97.61:40221)
I20260927 06:57:26.199801 13896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } attempt: 4
Sep 27 06:57:26 dist-test-slave-f892 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 06:57:26.216384 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:26.200641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48360 (local address 127.0.97.60:33977)
0927 06:57:26.200824 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:26.200830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:26.200847 (+ 17us) server_negotiation.cc:407] Connection header received
0927 06:57:26.200903 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:26.200908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:26.200991 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:26.201097 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:26.201927 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.203144 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.204914 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.205594 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.205735 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:26.209966 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:26.209972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:26.209977 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:26.210040 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:26.213528 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.214288 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.214295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.214299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.214458 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.214882 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.214887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.214891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.215272 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:26.215413 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:26.215816 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 06:57:26.216031 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 06:57:26.216416 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:26.200415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48360)
0927 06:57:26.200580 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:26.200601 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:26.200706 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:26.201125 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:26.201128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:26.201581 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:26.201787 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:26.201798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.203277 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.203284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:26.204762 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:26.204778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.205745 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.205751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:26.205940 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:26.206691 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:26.206715 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:26.209758 (+ 3043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:26.213648 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:26.213655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:26.213659 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:26.214164 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:26.214585 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:26.214589 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:26.214593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:26.214761 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:26.215375 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:26.215380 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:26.215560 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 06:57:26.215703 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 06:57:26.216001 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 06:57:26.222553 13896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595: Bootstrap starting.
I20260927 06:57:26.226583 13896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595: Neither blocks nor log segments found. Creating new log.
I20260927 06:57:26.227910 13896 log.cc:824] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:26.231070 13896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595: No bootstrap required, opened a new log
I20260927 06:57:26.241914 13896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } }
I20260927 06:57:26.242321 13896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:26.242427 13896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747819f0ab144ec1bb521fe514e08595, State: Initialized, Role: FOLLOWER
I20260927 06:57:26.243050 13896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } }
I20260927 06:57:26.244022 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } } }
I20260927 06:57:26.244428 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:57:26.245253 13896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:57:26.245311 13871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8459be6adaf449386b9adb1a3a3d005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747819f0ab144ec1bb521fe514e08595" is_pre_election: true
I20260927 06:57:26.245875 13871 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8459be6adaf449386b9adb1a3a3d005 in term 0.
I20260927 06:57:26.246587 13995 leader_election.cc:302] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 747819f0ab144ec1bb521fe514e08595, f8459be6adaf449386b9adb1a3a3d005; no voters:
I20260927 06:57:26.247061 14057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:57:26.247215 14057 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:57:26.247347 14057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:26.250703 14057 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } }
I20260927 06:57:26.251767 13871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8459be6adaf449386b9adb1a3a3d005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747819f0ab144ec1bb521fe514e08595"
I20260927 06:57:26.251921 14057 leader_election.cc:288] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [CANDIDATE]: Term 1 election: Requested vote from peers 747819f0ab144ec1bb521fe514e08595 (127.0.97.62:36295), ddd6ad857f9d4f888a12c54ad3cc9246 (127.0.97.61:40221)
I20260927 06:57:26.251961 13871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:26.255172 13871 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8459be6adaf449386b9adb1a3a3d005 in term 1.
I20260927 06:57:26.255740 13995 leader_election.cc:302] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 747819f0ab144ec1bb521fe514e08595, f8459be6adaf449386b9adb1a3a3d005; no voters:
I20260927 06:57:26.257736 14057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:57:26.258494 14057 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [term 1 LEADER]: Becoming Leader. State: Replica: f8459be6adaf449386b9adb1a3a3d005, State: Running, Role: LEADER
I20260927 06:57:26.259035 14057 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } }
I20260927 06:57:26.262421 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8459be6adaf449386b9adb1a3a3d005. Latest consensus state: current_term: 1 leader_uuid: "f8459be6adaf449386b9adb1a3a3d005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } } }
I20260927 06:57:26.262841 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [sys.catalog]: This master's current role is: LEADER
I20260927 06:57:26.264199 14079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:57:26.265307 13970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } attempt: 5
I20260927 06:57:26.269022 14079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 06:57:26.271037 13998 tablet.cc:2369] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 06:57:26 dist-test-slave-f892 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 06:57:26.285056 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:26.265871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33977 (local address 127.0.0.1:48368)
0927 06:57:26.266210 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:26.266230 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:26.266338 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:26.267218 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:26.267223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:26.267781 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:26.267989 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:26.268003 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.269598 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.269603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:26.271090 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:26.271107 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.272331 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.272346 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:26.272554 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:26.273245 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:26.273272 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:26.277087 (+ 3815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:26.281661 (+ 4574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:26.281669 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:26.281675 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:26.282293 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:26.282817 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:26.282825 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:26.282830 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:26.283008 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:26.283833 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:26.283838 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:26.284035 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 06:57:26.284194 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 06:57:26.284553 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206}
I20260927 06:57:26.286407 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:26.266553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.0.97.60:33977)
0927 06:57:26.266722 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:26.266733 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:26.266752 (+ 19us) server_negotiation.cc:407] Connection header received
0927 06:57:26.266813 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:26.266824 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:26.266915 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:26.267032 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:26.268164 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.269360 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.271265 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:26.271988 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:26.272113 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:26.277312 (+ 5199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:26.277323 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:26.277331 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:26.277369 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:26.281483 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.282451 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.282460 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.282466 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.282643 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:26.283157 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:26.283165 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:26.283170 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:26.283586 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:26.283751 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:26.285585 (+ 1834us) server_negotiation.cc:299] Negotiation successful
0927 06:57:26.285900 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 06:57:26.293066 13970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246: Bootstrap starting.
I20260927 06:57:26.298821 13970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246: Neither blocks nor log segments found. Creating new log.
W20260927 06:57:26.298995 13850 tablet.cc:2369] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:57:26.301671 13970 log.cc:824] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:26.306342 14086 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 06:57:26.310834 13970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246: No bootstrap required, opened a new log
I20260927 06:57:26.315970 13871 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595 [term 1 FOLLOWER]: Refusing update from remote peer f8459be6adaf449386b9adb1a3a3d005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:57:26.317003 14077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:57:26.322148 13970 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } }
I20260927 06:57:26.322511 13970 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:26.322657 13970 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddd6ad857f9d4f888a12c54ad3cc9246, State: Initialized, Role: FOLLOWER
I20260927 06:57:26.323390 13970 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } }
I20260927 06:57:26.311313 13829 master_runner.cc:186] Error getting master registration for 127.0.97.61:40221: OK, instance_id { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" instance_seqno: 1790492245540651 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 06:57:26.325682 13829 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 06:57:26.330458 14089 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } } }
I20260927 06:57:26.332508 14089 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:57:26.332408 13970 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:57:26.333421 13944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8459be6adaf449386b9adb1a3a3d005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246"
I20260927 06:57:26.333926 13944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:26.339295 13944 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8459be6adaf449386b9adb1a3a3d005 in term 1.
I20260927 06:57:26.334564 13945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8459be6adaf449386b9adb1a3a3d005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" is_pre_election: true
I20260927 06:57:26.341383 13943 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246 [term 1 FOLLOWER]: Refusing update from remote peer f8459be6adaf449386b9adb1a3a3d005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:57:26.343479 14077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:57:26.347474 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8459be6adaf449386b9adb1a3a3d005. Latest consensus state: current_term: 1 leader_uuid: "f8459be6adaf449386b9adb1a3a3d005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } } }
I20260927 06:57:26.347790 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:57:26.353493 14057 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8459be6adaf449386b9adb1a3a3d005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } } }
I20260927 06:57:26.353732 14079 catalog_manager.cc:1347] Generated new cluster ID: a5ad8a6afd714488913bfddebc76d5fa
I20260927 06:57:26.353832 14079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:57:26.353829 14057 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [sys.catalog]: This master's current role is: LEADER
I20260927 06:57:26.362583 14083 mvcc.cc:205] Tried to move back new op lower bound from 7333856240877375488 to 7333856240681676800. Current Snapshot: MvccSnapshot[applied={T|T < 7333856240877375488 or (T in {7333856240877375488})}]
I20260927 06:57:26.362823 14089 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8459be6adaf449386b9adb1a3a3d005. Latest consensus state: current_term: 1 leader_uuid: "f8459be6adaf449386b9adb1a3a3d005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } } }
I20260927 06:57:26.363119 14089 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:57:26.363876 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8459be6adaf449386b9adb1a3a3d005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } } }
I20260927 06:57:26.364147 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:57:26.366829 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8459be6adaf449386b9adb1a3a3d005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } } }
I20260927 06:57:26.367056 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005 [sys.catalog]: This master's current role is: LEADER
I20260927 06:57:26.373065 14098 mvcc.cc:205] Tried to move back new op lower bound from 7333856240877375488 to 7333856240681676800. Current Snapshot: MvccSnapshot[applied={T|T < 7333856240877375488}]
I20260927 06:57:26.373699 14089 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8459be6adaf449386b9adb1a3a3d005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747819f0ab144ec1bb521fe514e08595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36295 } } peers { permanent_uuid: "ddd6ad857f9d4f888a12c54ad3cc9246" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40221 } } peers { permanent_uuid: "f8459be6adaf449386b9adb1a3a3d005" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33977 } } }
I20260927 06:57:26.373934 14089 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:57:26.385787 14079 catalog_manager.cc:1370] Generated new certificate authority record
W20260927 06:57:26.387213 14106 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 06:57:26.387427 14106 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 06:57:26.387447 14079 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:57:26.400520 14079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8459be6adaf449386b9adb1a3a3d005: Generated new TSK 0
I20260927 06:57:26.401376 14079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:57:27.310549 14086 catalog_manager.cc:1259] Loaded cluster ID: a5ad8a6afd714488913bfddebc76d5fa
I20260927 06:57:27.310690 14086 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595: loading cluster ID for follower catalog manager: success
I20260927 06:57:27.313787 14086 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595: acquiring CA information for follower catalog manager: success
I20260927 06:57:27.315873 14086 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 747819f0ab144ec1bb521fe514e08595: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 06:57:27.389178 14106 catalog_manager.cc:1259] Loaded cluster ID: a5ad8a6afd714488913bfddebc76d5fa
I20260927 06:57:27.389274 14106 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246: loading cluster ID for follower catalog manager: success
I20260927 06:57:27.391746 14106 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246: acquiring CA information for follower catalog manager: success
I20260927 06:57:27.393452 14106 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ddd6ad857f9d4f888a12c54ad3cc9246: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 06:57:27.502168 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 13829
I20260927 06:57:27.503506 13891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:27.608520 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 13829
I20260927 06:57:27.627020 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 13903
I20260927 06:57:27.628103 13965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:27.733405 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 13903
I20260927 06:57:27.752985 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 13977
I20260927 06:57:27.753957 14039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:27.859514 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 13977
2026-09-27T06:57:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4520 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:57:27 dist-test-slave-f892 krb5kdc[14118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:57:27 dist-test-slave-f892 krb5kdc[14118](info): set up 2 sockets
Sep 27 06:57:27 dist-test-slave-f892 krb5kdc[14118](info): commencing operation
krb5kdc: starting...
W20260927 06:57:28.942487 388 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:57:29 dist-test-slave-f892 krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:57:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:57:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:57:29.153574 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43903
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:57:29.461992 14131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:29.462280 14131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:29.462468 14131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:29.469447 14131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:57:29.469542 14131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:29.469624 14131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:29.469727 14131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:57:29.469812 14131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:57:29.479351 14131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:29.480592 14131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:29.481808 14131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:29.492974 14137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:29.493048 14139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:29.493070 14136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:29.494131 14131 server_base.cc:1034] running on GCE node
I20260927 06:57:29.494858 14131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:29.496122 14131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:29.497319 14131 hybrid_clock.cc:648] HybridClock initialized: now 1790492249497280 us; error 36 us; skew 500 ppm
I20260927 06:57:29.497659 14131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:29 dist-test-slave-f892 krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:29.500602 14131 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:57:29.502090 14131 webserver.cc:460] Webserver started at http://127.0.97.62:42617/ using document root <none> and password file <none>
I20260927 06:57:29.502748 14131 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:29.502877 14131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:29.503193 14131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:57:29.506301 14131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "cd21930c0d714fdabb173d12b02ffdfe"
format_stamp: "Formatted at 2026-09-27 06:57:29 on dist-test-slave-f892"
server_key: "a998b51b5149c6dc1de78b37b34643c5"
server_key_iv: "1750b719e1609d529da5c5ede7b1e3b6"
server_key_version: "encryptionkey@0"
I20260927 06:57:29.507102 14131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "cd21930c0d714fdabb173d12b02ffdfe"
format_stamp: "Formatted at 2026-09-27 06:57:29 on dist-test-slave-f892"
server_key: "a998b51b5149c6dc1de78b37b34643c5"
server_key_iv: "1750b719e1609d529da5c5ede7b1e3b6"
server_key_version: "encryptionkey@0"
I20260927 06:57:29.512159 14131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 06:57:29.515542 14146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:29.517228 14131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 06:57:29.517421 14131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "cd21930c0d714fdabb173d12b02ffdfe"
format_stamp: "Formatted at 2026-09-27 06:57:29 on dist-test-slave-f892"
server_key: "a998b51b5149c6dc1de78b37b34643c5"
server_key_iv: "1750b719e1609d529da5c5ede7b1e3b6"
server_key_version: "encryptionkey@0"
I20260927 06:57:29.517657 14131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:29.537804 14131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:29.541576 14131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:29.541872 14131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:29.560194 14131 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43903
I20260927 06:57:29.560235 14197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43903 every 8 connection(s)
I20260927 06:57:29.561546 14131 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:57:29.562568 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 14131
I20260927 06:57:29.562811 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:57:29.563643 388 external_mini_cluster.cc:1428] Setting key 83b29f317b63ecf637cda11d996c69ef
I20260927 06:57:29.566282 14198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:29.576903 14198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe: Bootstrap starting.
Sep 27 06:57:29 dist-test-slave-f892 krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:29.580183 14198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe: Neither blocks nor log segments found. Creating new log.
I20260927 06:57:29.581377 14198 log.cc:824] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:29.584125 14198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe: No bootstrap required, opened a new log
I20260927 06:57:29.586663 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:29.565478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.0.97.62:43903)
0927 06:57:29.566563 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:29.566581 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:29.566644 (+ 63us) server_negotiation.cc:407] Connection header received
0927 06:57:29.568401 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:29.568421 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:29.568824 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:29.569302 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:29.570964 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:29.572262 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:29.573611 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:29.574365 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:29.574605 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:29.579119 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:29.579126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:29.579139 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:29.579172 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:29.582658 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:29.583318 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:29.583325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:29.583330 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:29.583490 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:29.583873 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:29.583879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:29.583882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:29.584428 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:29.584769 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:29.585196 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 06:57:29.585629 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":475,"threads_started":1}
I20260927 06:57:29.590314 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:57:29.592063 14198 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd21930c0d714fdabb173d12b02ffdfe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43903 } }
I20260927 06:57:29.592466 14198 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:29.592602 14198 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd21930c0d714fdabb173d12b02ffdfe, State: Initialized, Role: FOLLOWER
I20260927 06:57:29.593161 14198 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd21930c0d714fdabb173d12b02ffdfe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43903 } }
I20260927 06:57:29.593407 14198 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:57:29.593545 14198 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:57:29.593670 14198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:29.595659 14198 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd21930c0d714fdabb173d12b02ffdfe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43903 } }
I20260927 06:57:29.596071 14198 leader_election.cc:302] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd21930c0d714fdabb173d12b02ffdfe; no voters:
I20260927 06:57:29.596585 14198 leader_election.cc:288] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:57:29.596782 14203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:57:29.597427 14203 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe [term 1 LEADER]: Becoming Leader. State: Replica: cd21930c0d714fdabb173d12b02ffdfe, State: Running, Role: LEADER
I20260927 06:57:29.597901 14203 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd21930c0d714fdabb173d12b02ffdfe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43903 } }
I20260927 06:57:29.598578 14198 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:57:29.600698 14205 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd21930c0d714fdabb173d12b02ffdfe. Latest consensus state: current_term: 1 leader_uuid: "cd21930c0d714fdabb173d12b02ffdfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd21930c0d714fdabb173d12b02ffdfe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43903 } } }
I20260927 06:57:29.601152 14205 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe [sys.catalog]: This master's current role is: LEADER
I20260927 06:57:29.604310 14204 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd21930c0d714fdabb173d12b02ffdfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd21930c0d714fdabb173d12b02ffdfe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43903 } } }
I20260927 06:57:29.604739 14204 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe [sys.catalog]: This master's current role is: LEADER
W20260927 06:57:29.607594 14219 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 06:57:29.607723 14219 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 06:57:29.607861 14212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:57:29.609372 14212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:57:29.620995 14212 catalog_manager.cc:1347] Generated new cluster ID: d218afd054a0463897d7d0706a6cf146
I20260927 06:57:29.621099 14212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:57:29.633435 14212 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:57:29.635216 14212 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:57:29.649894 14212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd21930c0d714fdabb173d12b02ffdfe: Generated new TSK 0
I20260927 06:57:29.650926 14212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:57:29.702167 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 14131
I20260927 06:57:29.703320 14193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:29.797840 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 14131
2026-09-27T06:57:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1933 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:57:29 dist-test-slave-f892 krb5kdc[14226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:57:29 dist-test-slave-f892 krb5kdc[14226](info): set up 2 sockets
Sep 27 06:57:29 dist-test-slave-f892 krb5kdc[14226](info): commencing operation
krb5kdc: starting...
W20260927 06:57:30.906396 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:57:31 dist-test-slave-f892 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:57:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:57:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:57:31.119063 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36617
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:57:31.427841 14239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:31.428261 14239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:31.428413 14239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:31.435945 14239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:57:31.436087 14239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:31.436185 14239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:31.436283 14239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:57:31.436357 14239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:57:31.445772 14239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:31.447052 14239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:31.448287 14239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:31.459703 14244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:31.459735 14245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:31.459934 14247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:31.460740 14239 server_base.cc:1034] running on GCE node
I20260927 06:57:31.461390 14239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:31.462714 14239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:31.463932 14239 hybrid_clock.cc:648] HybridClock initialized: now 1790492251463893 us; error 49 us; skew 500 ppm
I20260927 06:57:31.464262 14239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:31 dist-test-slave-f892 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:31.472265 14239 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:57:31.473691 14239 webserver.cc:460] Webserver started at http://127.0.97.62:41467/ using document root <none> and password file <none>
I20260927 06:57:31.474418 14239 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:31.474565 14239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:31.474864 14239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:57:31.477730 14239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "e64894159daa4058a93c6b59fe0d3069"
format_stamp: "Formatted at 2026-09-27 06:57:31 on dist-test-slave-f892"
server_key: "0c635fdc60d0de862586a0e39d353139"
server_key_iv: "46e513a5080467ed853ecb1fbf0da697"
server_key_version: "encryptionkey@0"
I20260927 06:57:31.478469 14239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "e64894159daa4058a93c6b59fe0d3069"
format_stamp: "Formatted at 2026-09-27 06:57:31 on dist-test-slave-f892"
server_key: "0c635fdc60d0de862586a0e39d353139"
server_key_iv: "46e513a5080467ed853ecb1fbf0da697"
server_key_version: "encryptionkey@0"
I20260927 06:57:31.483294 14239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 06:57:31.486409 14254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:31.487640 14239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 06:57:31.487761 14239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "e64894159daa4058a93c6b59fe0d3069"
format_stamp: "Formatted at 2026-09-27 06:57:31 on dist-test-slave-f892"
server_key: "0c635fdc60d0de862586a0e39d353139"
server_key_iv: "46e513a5080467ed853ecb1fbf0da697"
server_key_version: "encryptionkey@0"
I20260927 06:57:31.487905 14239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:31.506382 14239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:31.510424 14239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:31.510643 14239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:31.528301 14239 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36617
I20260927 06:57:31.528337 14305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36617 every 8 connection(s)
I20260927 06:57:31.529546 14239 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:57:31.532930 14306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:31.538494 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 14239
I20260927 06:57:31.538692 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:57:31.539299 388 external_mini_cluster.cc:1428] Setting key 264975f64afaf4ac0fac8ac9b71f1b13
I20260927 06:57:31.542923 14306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069: Bootstrap starting.
I20260927 06:57:31.546473 14306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069: Neither blocks nor log segments found. Creating new log.
I20260927 06:57:31.547415 14306 log.cc:824] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:31.549957 14306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069: No bootstrap required, opened a new log
Sep 27 06:57:31 dist-test-slave-f892 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:31.557600 14306 raft_consensus.cc:348] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64894159daa4058a93c6b59fe0d3069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36617 } }
I20260927 06:57:31.558055 14306 raft_consensus.cc:374] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:31.558218 14306 raft_consensus.cc:729] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e64894159daa4058a93c6b59fe0d3069, State: Initialized, Role: FOLLOWER
I20260927 06:57:31.558822 14306 consensus_queue.cc:260] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64894159daa4058a93c6b59fe0d3069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36617 } }
I20260927 06:57:31.559096 14306 raft_consensus.cc:388] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:57:31.559242 14306 raft_consensus.cc:482] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:57:31.559432 14306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:31.560665 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:31.541215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.0.97.62:36617)
0927 06:57:31.541924 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:31.541934 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:31.541970 (+ 36us) server_negotiation.cc:407] Connection header received
0927 06:57:31.543656 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:31.543676 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:31.544005 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:31.544302 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:31.546141 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:31.547411 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:31.548722 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:31.549384 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:31.549589 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:31.553378 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:31.553385 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:31.553400 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:31.553445 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:31.556508 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:31.557129 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:31.557141 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:31.557146 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:31.557301 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:31.557719 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:31.557726 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:31.557729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:31.558445 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:31.558731 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:31.559237 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 06:57:31.559695 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":342,"threads_started":1}
I20260927 06:57:31.561771 14306 raft_consensus.cc:504] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64894159daa4058a93c6b59fe0d3069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36617 } }
I20260927 06:57:31.562322 14306 leader_election.cc:302] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e64894159daa4058a93c6b59fe0d3069; no voters:
I20260927 06:57:31.562942 14306 leader_election.cc:288] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:57:31.563159 14311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:57:31.563817 14311 raft_consensus.cc:686] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 1 LEADER]: Becoming Leader. State: Replica: e64894159daa4058a93c6b59fe0d3069, State: Running, Role: LEADER
I20260927 06:57:31.564301 14311 consensus_queue.cc:237] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64894159daa4058a93c6b59fe0d3069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36617 } }
I20260927 06:57:31.565698 14306 sys_catalog.cc:564] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:57:31.567255 14313 sys_catalog.cc:455] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e64894159daa4058a93c6b59fe0d3069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64894159daa4058a93c6b59fe0d3069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36617 } } }
I20260927 06:57:31.567214 14312 sys_catalog.cc:455] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e64894159daa4058a93c6b59fe0d3069. Latest consensus state: current_term: 1 leader_uuid: "e64894159daa4058a93c6b59fe0d3069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64894159daa4058a93c6b59fe0d3069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36617 } } }
I20260927 06:57:31.567566 14313 sys_catalog.cc:458] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [sys.catalog]: This master's current role is: LEADER
I20260927 06:57:31.567646 14312 sys_catalog.cc:458] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [sys.catalog]: This master's current role is: LEADER
I20260927 06:57:31.571009 14316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:57:31.575186 14316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:57:31.582952 14316 catalog_manager.cc:1347] Generated new cluster ID: bff8cdfe56fe470ea385bef9b9b3f970
I20260927 06:57:31.583048 14316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:57:31.593502 14316 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:57:31.595188 14316 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:57:31.604521 14316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069: Generated new TSK 0
I20260927 06:57:31.605373 14316 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 06:57:31.651752 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36617
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:57:31.964848 14332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:31.965118 14332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:31.965216 14332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:31.972127 14332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:31.972224 14332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:31.972333 14332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:57:31.981391 14332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:31.982519 14332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:31.983906 14332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:31.996625 14337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:31.996670 14338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:31.996712 14340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:31.997634 14332 server_base.cc:1034] running on GCE node
I20260927 06:57:31.998202 14332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:31.998970 14332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:32.000156 14332 hybrid_clock.cc:648] HybridClock initialized: now 1790492252000113 us; error 29 us; skew 500 ppm
I20260927 06:57:32.000463 14332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:31 dist-test-slave-f892 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:32.003328 14332 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:57:32.004626 14332 webserver.cc:460] Webserver started at http://127.0.97.1:42303/ using document root <none> and password file <none>
I20260927 06:57:32.005357 14332 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:32.005530 14332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:32.005815 14332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:57:32.008544 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "01c30625ed884c11959092b7148a8239"
format_stamp: "Formatted at 2026-09-27 06:57:32 on dist-test-slave-f892"
server_key: "fd770bcfb03fbfcd13740ed33ca7b0e9"
server_key_iv: "d1a1e42f4e17c6694cf196d2aa6c4397"
server_key_version: "encryptionkey@0"
I20260927 06:57:32.009263 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "01c30625ed884c11959092b7148a8239"
format_stamp: "Formatted at 2026-09-27 06:57:32 on dist-test-slave-f892"
server_key: "fd770bcfb03fbfcd13740ed33ca7b0e9"
server_key_iv: "d1a1e42f4e17c6694cf196d2aa6c4397"
server_key_version: "encryptionkey@0"
I20260927 06:57:32.014104 14332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.005s
I20260927 06:57:32.017195 14347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:32.018388 14332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 06:57:32.018556 14332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "01c30625ed884c11959092b7148a8239"
format_stamp: "Formatted at 2026-09-27 06:57:32 on dist-test-slave-f892"
server_key: "fd770bcfb03fbfcd13740ed33ca7b0e9"
server_key_iv: "d1a1e42f4e17c6694cf196d2aa6c4397"
server_key_version: "encryptionkey@0"
I20260927 06:57:32.018736 14332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:32.029878 14332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:32.033890 14332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:32.034204 14332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:32.035122 14332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:57:32.036347 14332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:57:32.036444 14332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:32.036599 14332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:57:32.036675 14332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:32.068388 14332 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40123
I20260927 06:57:32.068394 14459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40123 every 8 connection(s)
I20260927 06:57:32.069643 14332 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:57:32.070820 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 14332
I20260927 06:57:32.071012 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:57:32.071679 388 external_mini_cluster.cc:1428] Setting key d75d21e59a1595e7395e24f9168d9ac3
Sep 27 06:57:32 dist-test-slave-f892 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:32.092916 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:32.072175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37435 (local address 127.0.97.62:36617)
0927 06:57:32.074697 (+ 2522us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:32.074701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:32.074718 (+ 17us) server_negotiation.cc:407] Connection header received
0927 06:57:32.076754 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:32.076758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:32.076817 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:32.076903 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:32.078931 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:32.079624 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:32.081892 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:32.082677 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:32.082827 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:32.086811 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:32.086816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:32.086819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:32.086849 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:32.089426 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:32.090352 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:32.090355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:32.090357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:32.090465 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:32.090878 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:32.090887 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:32.090889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:32.091173 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:32.091329 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:32.092267 (+ 938us) server_negotiation.cc:299] Negotiation successful
0927 06:57:32.092502 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2424,"thread_start_us":2325,"threads_started":1}
I20260927 06:57:32.094213 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:32.072751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:37435)
0927 06:57:32.073839 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:32.073893 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:32.076497 (+ 2604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:32.077185 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:32.077199 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:32.077932 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:32.078751 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:32.078777 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:32.079783 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:32.079790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:32.081750 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:32.081762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:32.082876 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:32.082881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:32.083128 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:32.083863 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:32.083883 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:32.086597 (+ 2714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:32.089560 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:32.089571 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:32.089588 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:32.090238 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:32.090575 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:32.090582 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:32.090586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:32.090758 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:32.091329 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:32.091340 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:32.091810 (+ 470us) client_negotiation.cc:769] Sending connection context
0927 06:57:32.092151 (+ 341us) client_negotiation.cc:240] Negotiation successful
0927 06:57:32.092723 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":543,"threads_started":1}
I20260927 06:57:32.095723 14460 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36617
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 06:57:32.096046 14460 heartbeater.cc:464] Registering TS with master...
I20260927 06:57:32.096714 14460 heartbeater.cc:511] Master 127.0.97.62:36617 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 06:57:32.098973 14271 ts_manager.cc:194] Registered new tserver with Master: 01c30625ed884c11959092b7148a8239 (127.0.97.1:40123)
I20260927 06:57:32.101050 14271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 06:57:32.125247 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36617
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:57:32.432971 14466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:32.433259 14466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:32.433419 14466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:32.440446 14466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:32.440590 14466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:32.440747 14466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:57:32.449965 14466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:32.451189 14466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:32.452406 14466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:32.464673 14474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:32.464812 14472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:32.464993 14471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:32.465745 14466 server_base.cc:1034] running on GCE node
I20260927 06:57:32.466364 14466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:32.467208 14466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:32.468484 14466 hybrid_clock.cc:648] HybridClock initialized: now 1790492252468422 us; error 48 us; skew 500 ppm
I20260927 06:57:32.468832 14466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:32 dist-test-slave-f892 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:32.472486 14466 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:57:32.473732 14466 webserver.cc:460] Webserver started at http://127.0.97.2:36797/ using document root <none> and password file <none>
I20260927 06:57:32.474395 14466 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:32.474500 14466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:32.474730 14466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:57:32.477397 14466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "8931a6ee797b43d291c219d5f0b1404e"
format_stamp: "Formatted at 2026-09-27 06:57:32 on dist-test-slave-f892"
server_key: "b70d291a77ba8e941028dcf46c0c3a88"
server_key_iv: "0fd8399e10c377b9e50c9214d2abda32"
server_key_version: "encryptionkey@0"
I20260927 06:57:32.478169 14466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8931a6ee797b43d291c219d5f0b1404e"
format_stamp: "Formatted at 2026-09-27 06:57:32 on dist-test-slave-f892"
server_key: "b70d291a77ba8e941028dcf46c0c3a88"
server_key_iv: "0fd8399e10c377b9e50c9214d2abda32"
server_key_version: "encryptionkey@0"
I20260927 06:57:32.483304 14466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 06:57:32.486699 14481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:32.487897 14466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 06:57:32.488071 14466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "8931a6ee797b43d291c219d5f0b1404e"
format_stamp: "Formatted at 2026-09-27 06:57:32 on dist-test-slave-f892"
server_key: "b70d291a77ba8e941028dcf46c0c3a88"
server_key_iv: "0fd8399e10c377b9e50c9214d2abda32"
server_key_version: "encryptionkey@0"
I20260927 06:57:32.488267 14466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:32.514591 14466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:32.518399 14466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:32.518664 14466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:32.519526 14466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:57:32.520797 14466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:57:32.520906 14466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:32.521049 14466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:57:32.521121 14466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:32.556272 14466 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44123
I20260927 06:57:32.556354 14593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44123 every 8 connection(s)
I20260927 06:57:32.557464 14466 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:57:32.565008 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 14466
I20260927 06:57:32.565186 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:57:32.565680 388 external_mini_cluster.cc:1428] Setting key 9d2703305d90a4be3a02f6de462610a2
Sep 27 06:57:32 dist-test-slave-f892 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:32.576525 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:32.559862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52921 (local address 127.0.97.62:36617)
0927 06:57:32.560042 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:32.560046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:32.561120 (+ 1074us) server_negotiation.cc:407] Connection header received
0927 06:57:32.562885 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:32.562889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:32.562971 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:32.563057 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:32.564358 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:32.565312 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:32.566924 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:32.567581 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:32.567704 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:32.571026 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:32.571031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:32.571034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:32.571064 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:32.573605 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:32.574369 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:32.574378 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:32.574380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:32.574513 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:32.574939 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:32.574950 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:32.574953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:32.575198 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:32.575361 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:32.575971 (+ 610us) server_negotiation.cc:299] Negotiation successful
0927 06:57:32.576205 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 06:57:32.577399 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:32.560181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:52921)
0927 06:57:32.560962 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:32.561004 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:32.562659 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:32.563234 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:32.563241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:32.563723 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:32.564223 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:32.564238 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:32.565450 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:32.565453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:32.566755 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:32.566782 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:32.567782 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:32.567790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:32.568004 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:32.568665 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:32.568685 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:32.570814 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:32.573748 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:32.573757 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:32.573771 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:32.574246 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:32.574634 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:32.574638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:32.574642 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:32.574819 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:32.575332 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:32.575339 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:32.575628 (+ 289us) client_negotiation.cc:769] Sending connection context
0927 06:57:32.575882 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 06:57:32.576346 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":371,"threads_started":1}
I20260927 06:57:32.578815 14594 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36617
I20260927 06:57:32.579103 14594 heartbeater.cc:464] Registering TS with master...
I20260927 06:57:32.579739 14594 heartbeater.cc:511] Master 127.0.97.62:36617 requested a full tablet report, sending...
I20260927 06:57:32.581128 14271 ts_manager.cc:194] Registered new tserver with Master: 8931a6ee797b43d291c219d5f0b1404e (127.0.97.2:44123)
I20260927 06:57:32.582410 14271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52921
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 06:57:32.614373 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36617
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:57:32.932945 14599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:32.933187 14599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:32.933295 14599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:32.940266 14599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:32.940363 14599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:32.940485 14599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:57:32.949575 14599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:32.950731 14599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:32.952008 14599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:32.964154 14604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:32.964128 14605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:32.964221 14607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:32.965157 14599 server_base.cc:1034] running on GCE node
I20260927 06:57:32.965689 14599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:32.966648 14599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:32.967830 14599 hybrid_clock.cc:648] HybridClock initialized: now 1790492252967794 us; error 32 us; skew 500 ppm
I20260927 06:57:32.968106 14599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:32 dist-test-slave-f892 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:32.971011 14599 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:57:32.972361 14599 webserver.cc:460] Webserver started at http://127.0.97.3:36713/ using document root <none> and password file <none>
I20260927 06:57:32.972972 14599 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:32.973109 14599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:32.973407 14599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:57:32.976099 14599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "454b008c6926463e9778ee64e8c06a84"
format_stamp: "Formatted at 2026-09-27 06:57:32 on dist-test-slave-f892"
server_key: "a73cdd61065142369eb0b2b5b6ca6503"
server_key_iv: "f1b2aeb534d38ed749f3b67bacf9fd88"
server_key_version: "encryptionkey@0"
I20260927 06:57:32.976809 14599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "454b008c6926463e9778ee64e8c06a84"
format_stamp: "Formatted at 2026-09-27 06:57:32 on dist-test-slave-f892"
server_key: "a73cdd61065142369eb0b2b5b6ca6503"
server_key_iv: "f1b2aeb534d38ed749f3b67bacf9fd88"
server_key_version: "encryptionkey@0"
I20260927 06:57:32.981338 14599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 06:57:32.984531 14614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:32.985685 14599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 06:57:32.985860 14599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "454b008c6926463e9778ee64e8c06a84"
format_stamp: "Formatted at 2026-09-27 06:57:32 on dist-test-slave-f892"
server_key: "a73cdd61065142369eb0b2b5b6ca6503"
server_key_iv: "f1b2aeb534d38ed749f3b67bacf9fd88"
server_key_version: "encryptionkey@0"
I20260927 06:57:32.986073 14599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:33.007437 14599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:33.010946 14599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:33.011204 14599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:33.012091 14599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:57:33.013396 14599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:57:33.013496 14599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:33.013650 14599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:57:33.013720 14599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:33.045077 14599 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44137
I20260927 06:57:33.045123 14726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44137 every 8 connection(s)
I20260927 06:57:33.046303 14599 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:57:33.053467 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 14599
I20260927 06:57:33.053654 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:57:33.054154 388 external_mini_cluster.cc:1428] Setting key 8d16f74b2c7b681cb49a989f9ce04f29
Sep 27 06:57:33 dist-test-slave-f892 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:33.065052 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:33.048641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39771 (local address 127.0.97.62:36617)
0927 06:57:33.048837 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:33.048843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:33.049935 (+ 1092us) server_negotiation.cc:407] Connection header received
0927 06:57:33.051704 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:33.051709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:33.051774 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:33.051859 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:33.053160 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.053960 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.056067 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.056686 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.056825 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:33.060083 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:33.060088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:33.060091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:33.060139 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:33.062736 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:33.063333 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:33.063340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:33.063342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:33.063473 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:33.063806 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:33.063809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:33.063811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:33.064044 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:33.064155 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:33.064687 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 06:57:33.064830 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 06:57:33.065886 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:33.049110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:39771)
0927 06:57:33.049781 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:33.049819 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:33.051471 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:33.052038 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:33.052045 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:33.052499 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:33.053003 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:33.053022 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.054123 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.054129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:33.055910 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:33.055924 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.056939 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.056946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:33.057204 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:33.057795 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:33.057814 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:33.059900 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:33.062849 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:33.062855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:33.062864 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:33.063243 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:33.063564 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:33.063567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:33.063569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:33.063721 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:33.064141 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:33.064145 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:33.064387 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 06:57:33.064587 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 06:57:33.064900 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":279,"threads_started":1}
I20260927 06:57:33.067317 14727 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36617
I20260927 06:57:33.067624 14727 heartbeater.cc:464] Registering TS with master...
I20260927 06:57:33.068400 14727 heartbeater.cc:511] Master 127.0.97.62:36617 requested a full tablet report, sending...
I20260927 06:57:33.069849 14271 ts_manager.cc:194] Registered new tserver with Master: 454b008c6926463e9778ee64e8c06a84 (127.0.97.3:44137)
I20260927 06:57:33.070861 14271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39771
I20260927 06:57:33.080791 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:57:33.096673 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:33.085063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55428 (local address 127.0.97.62:36617)
0927 06:57:33.085209 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:33.085213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:33.085664 (+ 451us) server_negotiation.cc:407] Connection header received
0927 06:57:33.085788 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:33.085791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:33.085845 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:33.085919 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:33.087242 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.087911 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.089212 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.089573 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.089663 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:33.091933 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:33.091937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:33.091940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:33.091969 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:33.094626 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:33.095221 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:33.095224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:33.095226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:33.095334 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:33.095619 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:33.095622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:33.095624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:33.095881 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:33.096061 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:33.096311 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 06:57:33.096457 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 06:57:33.103848 14460 heartbeater.cc:503] Master 127.0.97.62:36617 was elected leader, sending a full tablet report...
I20260927 06:57:33.100121 14271 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:57:33.107148 14271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:57:33.128948 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:33.117848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44123 (local address 127.0.0.1:40456)
0927 06:57:33.118869 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:33.118906 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:33.119063 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:33.121037 (+ 1974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:33.121047 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:33.121079 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:33.121951 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:33.121962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.123746 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.123751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:33.126089 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:33.126106 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.127907 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.127913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:33.128146 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:33.128196 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 06:57:33.128311 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 06:57:33.128492 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":2,"thread_start_us":603,"threads_started":1}
I20260927 06:57:33.129465 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:33.118181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40456 (local address 127.0.97.2:44123)
0927 06:57:33.120272 (+ 2091us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:33.120282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:33.120316 (+ 34us) server_negotiation.cc:407] Connection header received
0927 06:57:33.120433 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:33.120444 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:33.120784 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:33.121269 (+ 485us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:33.122321 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.123610 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.126266 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.127713 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.127844 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:33.128019 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:33.128966 (+ 947us) server_negotiation.cc:299] Negotiation successful
0927 06:57:33.129161 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1899,"thread_start_us":1761,"threads_started":1}
I20260927 06:57:33.130121 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:33.119055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55738 (local address 127.0.97.1:40123)
0927 06:57:33.119784 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:33.119795 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:33.121584 (+ 1789us) server_negotiation.cc:407] Connection header received
0927 06:57:33.122322 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:33.122330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:33.122492 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:33.122618 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:33.123523 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.125016 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.127318 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.128499 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.128631 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:33.128837 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:33.129603 (+ 766us) server_negotiation.cc:299] Negotiation successful
0927 06:57:33.129773 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":382,"threads_started":1}
I20260927 06:57:33.130625 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:33.118521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40123 (local address 127.0.0.1:55738)
0927 06:57:33.121441 (+ 2920us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:33.121465 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:33.122085 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:33.122814 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:33.122817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:33.122835 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:33.123330 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:33.123343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.125184 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.125189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:33.127176 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:33.127187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.128709 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.128716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:33.128894 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:33.128935 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 06:57:33.130181 (+ 1246us) client_negotiation.cc:240] Negotiation successful
0927 06:57:33.130309 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2524,"thread_start_us":2357,"threads_started":1}
I20260927 06:57:33.131224 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:33.119218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44137 (local address 127.0.0.1:52014)
0927 06:57:33.123153 (+ 3935us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:33.123171 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:33.123306 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:33.123895 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:33.123899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:33.123924 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:33.124321 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:33.124331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.125966 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.125971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:33.128165 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:33.128176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.130106 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.130113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:33.130301 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:33.130332 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 06:57:33.130431 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 06:57:33.130551 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3808,"mutex_wait_us":27,"thread_start_us":2970,"threads_started":1}
I20260927 06:57:33.131242 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:33.119688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.0.97.3:44137)
0927 06:57:33.120708 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:33.120720 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:33.123280 (+ 2560us) server_negotiation.cc:407] Connection header received
0927 06:57:33.123475 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:33.123483 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:33.123710 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:33.123903 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:33.124487 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.125844 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.128307 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.129609 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.129737 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:33.129934 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:33.130747 (+ 813us) server_negotiation.cc:299] Negotiation successful
0927 06:57:33.130916 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":597,"threads_started":1}
I20260927 06:57:33.133848 14529 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad66e5e270d4e9f9cc2fd0216142b5d (DEFAULT_TABLE table=test-table [id=9ad7a98073b34fc2a800dd5e5815dc50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:57:33.134126 14395 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad66e5e270d4e9f9cc2fd0216142b5d (DEFAULT_TABLE table=test-table [id=9ad7a98073b34fc2a800dd5e5815dc50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:57:33.134294 14662 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad66e5e270d4e9f9cc2fd0216142b5d (DEFAULT_TABLE table=test-table [id=9ad7a98073b34fc2a800dd5e5815dc50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:57:33.135329 14395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad66e5e270d4e9f9cc2fd0216142b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:33.135351 14529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad66e5e270d4e9f9cc2fd0216142b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:33.135422 14662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad66e5e270d4e9f9cc2fd0216142b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:33.145798 14745 tablet_bootstrap.cc:492] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84: Bootstrap starting.
I20260927 06:57:33.149151 14746 tablet_bootstrap.cc:492] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e: Bootstrap starting.
I20260927 06:57:33.149701 14745 tablet_bootstrap.cc:654] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84: Neither blocks nor log segments found. Creating new log.
I20260927 06:57:33.150074 14747 tablet_bootstrap.cc:492] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239: Bootstrap starting.
I20260927 06:57:33.151059 14745 log.cc:824] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:33.152522 14746 tablet_bootstrap.cc:654] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e: Neither blocks nor log segments found. Creating new log.
I20260927 06:57:33.153200 14747 tablet_bootstrap.cc:654] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239: Neither blocks nor log segments found. Creating new log.
I20260927 06:57:33.153681 14746 log.cc:824] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:33.154202 14745 tablet_bootstrap.cc:492] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84: No bootstrap required, opened a new log
I20260927 06:57:33.154241 14747 log.cc:824] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:33.154536 14745 ts_tablet_manager.cc:1397] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 06:57:33.156520 14746 tablet_bootstrap.cc:492] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e: No bootstrap required, opened a new log
I20260927 06:57:33.156771 14746 ts_tablet_manager.cc:1397] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 06:57:33.157286 14747 tablet_bootstrap.cc:492] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239: No bootstrap required, opened a new log
I20260927 06:57:33.157552 14747 ts_tablet_manager.cc:1397] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260927 06:57:33.162367 14745 raft_consensus.cc:348] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } } peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } }
I20260927 06:57:33.162686 14745 raft_consensus.cc:374] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:33.162806 14745 raft_consensus.cc:729] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 454b008c6926463e9778ee64e8c06a84, State: Initialized, Role: FOLLOWER
I20260927 06:57:33.163410 14745 consensus_queue.cc:260] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } } peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } }
I20260927 06:57:33.164398 14727 heartbeater.cc:503] Master 127.0.97.62:36617 was elected leader, sending a full tablet report...
I20260927 06:57:33.165082 14745 ts_tablet_manager.cc:1428] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260927 06:57:33.167174 14746 raft_consensus.cc:348] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } } peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } }
I20260927 06:57:33.167588 14746 raft_consensus.cc:374] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:33.167697 14746 raft_consensus.cc:729] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8931a6ee797b43d291c219d5f0b1404e, State: Initialized, Role: FOLLOWER
I20260927 06:57:33.167654 14747 raft_consensus.cc:348] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } } peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } }
I20260927 06:57:33.168129 14747 raft_consensus.cc:374] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:33.168253 14747 raft_consensus.cc:729] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c30625ed884c11959092b7148a8239, State: Initialized, Role: FOLLOWER
I20260927 06:57:33.168351 14746 consensus_queue.cc:260] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } } peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } }
I20260927 06:57:33.168800 14747 consensus_queue.cc:260] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } } peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } }
I20260927 06:57:33.169467 14594 heartbeater.cc:503] Master 127.0.97.62:36617 was elected leader, sending a full tablet report...
I20260927 06:57:33.170348 14746 ts_tablet_manager.cc:1428] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260927 06:57:33.171983 14747 ts_tablet_manager.cc:1428] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239: Time spent starting tablet: real 0.014s user 0.007s sys 0.008s
W20260927 06:57:33.297948 14728 tablet.cc:2369] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:57:33.309441 14595 tablet.cc:2369] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:57:33.322136 14461 tablet.cc:2369] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:57:33.370313 14751 raft_consensus.cc:482] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:57:33.370600 14751 raft_consensus.cc:504] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } } peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } }
I20260927 06:57:33.372273 14751 leader_election.cc:288] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8931a6ee797b43d291c219d5f0b1404e (127.0.97.2:44123), 01c30625ed884c11959092b7148a8239 (127.0.97.1:40123)
I20260927 06:57:33.379355 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:33.372792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40039 (local address 127.0.97.1:40123)
0927 06:57:33.372979 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:33.372985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:33.373204 (+ 219us) server_negotiation.cc:407] Connection header received
0927 06:57:33.373423 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:33.373427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:33.373484 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:33.373585 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:33.373997 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.374967 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.377355 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.378569 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.378706 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:33.378812 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:33.379037 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 06:57:33.379141 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 06:57:33.379436 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:33.372701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40123 (local address 127.0.97.3:40039)
0927 06:57:33.373068 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:33.373091 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:33.373258 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:33.373617 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:33.373622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:33.373650 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:33.373837 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:33.373845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.375107 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.375110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:33.377210 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:33.377224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.378714 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.378718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:33.378851 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:33.378874 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 06:57:33.379003 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 06:57:33.379137 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":86}
I20260927 06:57:33.379740 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:33.372643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57473 (local address 127.0.97.2:44123)
0927 06:57:33.372848 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:33.372855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:33.373612 (+ 757us) server_negotiation.cc:407] Connection header received
0927 06:57:33.373761 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:33.373765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:33.373819 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:33.373887 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:33.374448 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.375330 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.377591 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.378814 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.378929 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:33.379032 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:33.379358 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 06:57:33.379506 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 06:57:33.379776 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:33.372759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44123 (local address 127.0.97.3:57473)
0927 06:57:33.373485 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:33.373501 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:33.373627 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:33.373935 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:33.373941 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:33.373966 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:33.374293 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:33.374300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.375471 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.375476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:33.377463 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:33.377473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.379031 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.379036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:33.379194 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:33.379226 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 06:57:33.379316 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 06:57:33.379418 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":387,"threads_started":1}
I20260927 06:57:33.380513 14415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad66e5e270d4e9f9cc2fd0216142b5d" candidate_uuid: "454b008c6926463e9778ee64e8c06a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c30625ed884c11959092b7148a8239" is_pre_election: true
I20260927 06:57:33.380638 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad66e5e270d4e9f9cc2fd0216142b5d" candidate_uuid: "454b008c6926463e9778ee64e8c06a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8931a6ee797b43d291c219d5f0b1404e" is_pre_election: true
I20260927 06:57:33.381019 14415 raft_consensus.cc:2454] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 454b008c6926463e9778ee64e8c06a84 in term 0.
I20260927 06:57:33.381122 14549 raft_consensus.cc:2454] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 454b008c6926463e9778ee64e8c06a84 in term 0.
I20260927 06:57:33.381752 14616 leader_election.cc:302] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 454b008c6926463e9778ee64e8c06a84, 8931a6ee797b43d291c219d5f0b1404e; no voters:
I20260927 06:57:33.382189 14751 raft_consensus.cc:2781] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:57:33.382347 14751 raft_consensus.cc:482] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:57:33.382457 14751 raft_consensus.cc:3037] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:33.384811 14751 raft_consensus.cc:504] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } } peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } }
I20260927 06:57:33.385397 14751 leader_election.cc:288] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [CANDIDATE]: Term 1 election: Requested vote from peers 8931a6ee797b43d291c219d5f0b1404e (127.0.97.2:44123), 01c30625ed884c11959092b7148a8239 (127.0.97.1:40123)
I20260927 06:57:33.385818 14415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad66e5e270d4e9f9cc2fd0216142b5d" candidate_uuid: "454b008c6926463e9778ee64e8c06a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c30625ed884c11959092b7148a8239"
I20260927 06:57:33.385812 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad66e5e270d4e9f9cc2fd0216142b5d" candidate_uuid: "454b008c6926463e9778ee64e8c06a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8931a6ee797b43d291c219d5f0b1404e"
I20260927 06:57:33.386013 14549 raft_consensus.cc:3037] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:33.386037 14415 raft_consensus.cc:3037] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:33.389097 14415 raft_consensus.cc:2454] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 454b008c6926463e9778ee64e8c06a84 in term 1.
I20260927 06:57:33.389097 14549 raft_consensus.cc:2454] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 454b008c6926463e9778ee64e8c06a84 in term 1.
I20260927 06:57:33.389496 14616 leader_election.cc:302] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 454b008c6926463e9778ee64e8c06a84, 8931a6ee797b43d291c219d5f0b1404e; no voters:
I20260927 06:57:33.389765 14751 raft_consensus.cc:2781] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:57:33.390220 14751 raft_consensus.cc:686] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [term 1 LEADER]: Becoming Leader. State: Replica: 454b008c6926463e9778ee64e8c06a84, State: Running, Role: LEADER
I20260927 06:57:33.390725 14751 consensus_queue.cc:237] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } } peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } }
I20260927 06:57:33.395025 14271 catalog_manager.cc:5495] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 454b008c6926463e9778ee64e8c06a84 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "454b008c6926463e9778ee64e8c06a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } health_report { overall_health: UNKNOWN } } }
I20260927 06:57:33.418342 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:33.413565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52030 (local address 127.0.97.3:44137)
0927 06:57:33.413758 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:33.413763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:33.413797 (+ 34us) server_negotiation.cc:407] Connection header received
0927 06:57:33.413973 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:33.413977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:33.414112 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:33.414227 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:57:33.414553 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.415226 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.416493 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.416849 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.416931 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:33.417357 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:57:33.417838 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:57:33.417942 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:33.418104 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 06:57:33.418162 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 06:57:33.425617 14415 raft_consensus.cc:1261] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Refusing update from remote peer 454b008c6926463e9778ee64e8c06a84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:57:33.425618 14549 raft_consensus.cc:1261] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e [term 1 FOLLOWER]: Refusing update from remote peer 454b008c6926463e9778ee64e8c06a84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:57:33.426636 14755 consensus_queue.cc:1035] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:57:33.427003 14751 consensus_queue.cc:1035] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:57:33.439385 14759 mvcc.cc:205] Tried to move back new op lower bound from 7333856270022877184 to 7333856269892481024. Current Snapshot: MvccSnapshot[applied={T|T < 7333856270022877184}]
I20260927 06:57:33.438972 14758 mvcc.cc:205] Tried to move back new op lower bound from 7333856270022877184 to 7333856269892481024. Current Snapshot: MvccSnapshot[applied={T|T < 7333856270022877184 or (T in {7333856270022877184})}]
I20260927 06:57:33.449110 14271 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:57:33.449359 14271 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:57:33.452857 14271 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069: Sending DeleteTablet for 3 replicas of tablet 3ad66e5e270d4e9f9cc2fd0216142b5d
I20260927 06:57:33.453883 14529 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ad66e5e270d4e9f9cc2fd0216142b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:57:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40456
I20260927 06:57:33.454020 14662 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ad66e5e270d4e9f9cc2fd0216142b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:57:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52014
I20260927 06:57:33.454229 14395 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ad66e5e270d4e9f9cc2fd0216142b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:57:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55738
I20260927 06:57:33.455085 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 14332
I20260927 06:57:33.479943 14770 tablet_replica.cc:330] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239: stopping tablet replica
I20260927 06:57:33.482373 14770 raft_consensus.cc:2229] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:57:33.484299 14769 tablet_replica.cc:330] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84: stopping tablet replica
I20260927 06:57:33.484541 14770 raft_consensus.cc:2258] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:57:33.484764 14769 raft_consensus.cc:2229] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:57:33.485426 14768 tablet_replica.cc:330] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e: stopping tablet replica
I20260927 06:57:33.485772 14768 raft_consensus.cc:2229] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:57:33.486248 14768 raft_consensus.cc:2258] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:57:33.487275 14769 raft_consensus.cc:2258] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:57:33.487790 14770 ts_tablet_manager.cc:1905] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:57:33.490437 14769 ts_tablet_manager.cc:1905] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:57:33.490844 14768 ts_tablet_manager.cc:1905] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:57:33.495020 14770 ts_tablet_manager.cc:1918] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:57:33.495168 14770 log.cc:1196] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/3ad66e5e270d4e9f9cc2fd0216142b5d
I20260927 06:57:33.495707 14770 ts_tablet_manager.cc:1939] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 01c30625ed884c11959092b7148a8239: Deleting consensus metadata
I20260927 06:57:33.497450 14768 ts_tablet_manager.cc:1918] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:57:33.497478 14257 catalog_manager.cc:4841] TS 01c30625ed884c11959092b7148a8239 (127.0.97.1:40123): tablet 3ad66e5e270d4e9f9cc2fd0216142b5d (table test-table [id=9ad7a98073b34fc2a800dd5e5815dc50]) successfully deleted
I20260927 06:57:33.497604 14768 log.cc:1196] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/3ad66e5e270d4e9f9cc2fd0216142b5d
I20260927 06:57:33.498162 14768 ts_tablet_manager.cc:1939] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 8931a6ee797b43d291c219d5f0b1404e: Deleting consensus metadata
Sep 27 06:57:33 dist-test-slave-f892 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:57:33.499704 14256 catalog_manager.cc:4841] TS 8931a6ee797b43d291c219d5f0b1404e (127.0.97.2:44123): tablet 3ad66e5e270d4e9f9cc2fd0216142b5d (table test-table [id=9ad7a98073b34fc2a800dd5e5815dc50]) successfully deleted
I20260927 06:57:33.501238 14769 ts_tablet_manager.cc:1918] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:57:33.501410 14769 log.cc:1196] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/3ad66e5e270d4e9f9cc2fd0216142b5d
I20260927 06:57:33.501814 14769 ts_tablet_manager.cc:1939] T 3ad66e5e270d4e9f9cc2fd0216142b5d P 454b008c6926463e9778ee64e8c06a84: Deleting consensus metadata
I20260927 06:57:33.503183 14256 catalog_manager.cc:4841] TS 454b008c6926463e9778ee64e8c06a84 (127.0.97.3:44137): tablet 3ad66e5e270d4e9f9cc2fd0216142b5d (table test-table [id=9ad7a98073b34fc2a800dd5e5815dc50]) successfully deleted
I20260927 06:57:33.507423 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:33.456315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55742 (local address 127.0.97.1:40123)
0927 06:57:33.456518 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:33.456526 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:33.478911 (+ 22385us) server_negotiation.cc:407] Connection header received
0927 06:57:33.480227 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:33.480236 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:33.480356 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:33.480450 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:33.485004 (+ 4554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.486130 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.489345 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.489945 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.490135 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:33.500561 (+ 10426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:33.500574 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:33.500591 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:33.500636 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:33.504354 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:33.505007 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:33.505015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:33.505019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:33.505168 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:33.505505 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:33.505509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:33.505513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:33.505927 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:33.506291 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:33.506785 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 06:57:33.507036 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 06:57:33.508137 14455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:33.596349 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 14332
I20260927 06:57:33.613248 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 14466
Sep 27 06:57:33 dist-test-slave-f892 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:57:33.630312 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:33.614153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40470 (local address 127.0.97.2:44123)
0927 06:57:33.614324 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:33.614328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:33.614346 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:57:33.614443 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:33.614447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:33.614524 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:33.614603 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:33.616201 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.616938 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.618969 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.619602 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.619730 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:33.623858 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:33.623878 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:33.623897 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:33.623935 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:33.627506 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:33.628136 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:33.628146 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:33.628149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:33.628289 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:33.628636 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:33.628641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:33.628645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:33.629123 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:33.629399 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:33.629685 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 06:57:33.629910 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 06:57:33.631090 14589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:33.723088 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 14466
I20260927 06:57:33.746313 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 14599
Sep 27 06:57:33 dist-test-slave-f892 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 06:57:33.763618 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:33.747063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.0.97.3:44137)
0927 06:57:33.747244 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:33.747251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:33.747285 (+ 34us) server_negotiation.cc:407] Connection header received
0927 06:57:33.747404 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:33.747411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:33.747501 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:33.747624 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:33.748832 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.750206 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.751518 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:33.752227 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:33.752352 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:33.756500 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:33.756513 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:33.756532 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:33.756575 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:33.760621 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:33.761324 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:33.761334 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:33.761337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:33.761544 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:33.761885 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:33.761891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:33.761895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:33.762384 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:33.762676 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:33.762970 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 06:57:33.763192 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 06:57:33.764389 14722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:33.882073 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 14599
I20260927 06:57:33.901018 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 14239
I20260927 06:57:33.902408 14301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:34.010300 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 14239
I20260927 06:57:34.031049 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36617
--webserver_interface=127.0.97.62
--webserver_port=41467
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:57:34.339013 14776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:34.339284 14776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:34.339471 14776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:34.346475 14776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:57:34.346590 14776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:34.346674 14776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:34.346772 14776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:57:34.346835 14776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:57:34.356609 14776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=41467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:34.357733 14776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:34.359390 14776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:34.380259 14783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:34.380465 14782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:34.380452 14785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:34.381331 14776 server_base.cc:1034] running on GCE node
I20260927 06:57:34.382341 14776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:34.383903 14776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:34.385102 14776 hybrid_clock.cc:648] HybridClock initialized: now 1790492254385052 us; error 36 us; skew 500 ppm
I20260927 06:57:34.385510 14776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:34 dist-test-slave-f892 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:34.388903 14776 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:57:34.390507 14776 webserver.cc:460] Webserver started at http://127.0.97.62:41467/ using document root <none> and password file <none>
I20260927 06:57:34.391238 14776 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:34.391399 14776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:34.397089 14776 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260927 06:57:34.399962 14792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:34.401191 14776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 06:57:34.401382 14776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "e64894159daa4058a93c6b59fe0d3069"
format_stamp: "Formatted at 2026-09-27 06:57:31 on dist-test-slave-f892"
server_key: "0c635fdc60d0de862586a0e39d353139"
server_key_iv: "46e513a5080467ed853ecb1fbf0da697"
server_key_version: "encryptionkey@0"
I20260927 06:57:34.402493 14776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:34.419620 14776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:34.423594 14776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:34.423880 14776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:34.442272 14776 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36617
I20260927 06:57:34.442294 14843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36617 every 8 connection(s)
I20260927 06:57:34.443809 14776 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:57:34.450485 14844 sys_catalog.cc:263] Verifying existing consensus state
I20260927 06:57:34.451220 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 14776
I20260927 06:57:34.452181 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40123
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:36617
--webserver_port=42303
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:57:34.455459 14844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069: Bootstrap starting.
I20260927 06:57:34.475845 14844 log.cc:824] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:34.487771 14844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 06:57:34.488324 14844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069: Bootstrap complete.
I20260927 06:57:34.497700 14844 raft_consensus.cc:348] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64894159daa4058a93c6b59fe0d3069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36617 } }
I20260927 06:57:34.498754 14844 raft_consensus.cc:729] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e64894159daa4058a93c6b59fe0d3069, State: Initialized, Role: FOLLOWER
I20260927 06:57:34.499411 14844 consensus_queue.cc:260] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64894159daa4058a93c6b59fe0d3069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36617 } }
I20260927 06:57:34.499642 14844 raft_consensus.cc:388] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:57:34.499752 14844 raft_consensus.cc:482] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:57:34.499934 14844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 1 FOLLOWER]: Advancing to term 2
I20260927 06:57:34.503523 14844 raft_consensus.cc:504] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64894159daa4058a93c6b59fe0d3069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36617 } }
I20260927 06:57:34.504051 14844 leader_election.cc:302] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e64894159daa4058a93c6b59fe0d3069; no voters:
I20260927 06:57:34.504684 14844 leader_election.cc:288] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 06:57:34.504825 14848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 06:57:34.505928 14848 raft_consensus.cc:686] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 2 LEADER]: Becoming Leader. State: Replica: e64894159daa4058a93c6b59fe0d3069, State: Running, Role: LEADER
I20260927 06:57:34.506654 14848 consensus_queue.cc:237] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64894159daa4058a93c6b59fe0d3069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36617 } }
I20260927 06:57:34.506917 14844 sys_catalog.cc:564] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:57:34.508915 14850 sys_catalog.cc:455] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e64894159daa4058a93c6b59fe0d3069. Latest consensus state: current_term: 2 leader_uuid: "e64894159daa4058a93c6b59fe0d3069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64894159daa4058a93c6b59fe0d3069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36617 } } }
I20260927 06:57:34.509277 14850 sys_catalog.cc:458] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [sys.catalog]: This master's current role is: LEADER
I20260927 06:57:34.509840 14849 sys_catalog.cc:455] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e64894159daa4058a93c6b59fe0d3069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64894159daa4058a93c6b59fe0d3069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36617 } } }
I20260927 06:57:34.510147 14849 sys_catalog.cc:458] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [sys.catalog]: This master's current role is: LEADER
I20260927 06:57:34.514858 14855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:57:34.524247 14855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:57:34.526402 14855 catalog_manager.cc:1259] Loaded cluster ID: bff8cdfe56fe470ea385bef9b9b3f970
I20260927 06:57:34.526552 14855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:57:34.530458 14855 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 06:57:34.556529 14855 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 06:57:34.568296 14855 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:57:34.570585 14855 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069: Loaded TSK: 0
I20260927 06:57:34.572508 14855 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 06:57:34.831413 14846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:34.831725 14846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:34.831888 14846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:34.839103 14846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:34.839255 14846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:34.839414 14846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:57:34.848953 14846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42303
--tserver_master_addrs=127.0.97.62:36617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:34.850174 14846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:34.851368 14846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:34.865185 14871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:34.865464 14872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:34.865898 14874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:34.866349 14846 server_base.cc:1034] running on GCE node
I20260927 06:57:34.866930 14846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:34.867839 14846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:34.869064 14846 hybrid_clock.cc:648] HybridClock initialized: now 1790492254869013 us; error 44 us; skew 500 ppm
I20260927 06:57:34.869362 14846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:34 dist-test-slave-f892 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:34.872550 14846 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:57:34.874121 14846 webserver.cc:460] Webserver started at http://127.0.97.1:42303/ using document root <none> and password file <none>
I20260927 06:57:34.874784 14846 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:34.874962 14846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:34.881235 14846 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 06:57:34.886166 14881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:34.887538 14846 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 06:57:34.887665 14846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "01c30625ed884c11959092b7148a8239"
format_stamp: "Formatted at 2026-09-27 06:57:32 on dist-test-slave-f892"
server_key: "fd770bcfb03fbfcd13740ed33ca7b0e9"
server_key_iv: "d1a1e42f4e17c6694cf196d2aa6c4397"
server_key_version: "encryptionkey@0"
I20260927 06:57:34.888569 14846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:34.908357 14846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:34.912534 14846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:34.912848 14846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:34.913875 14846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:57:34.915335 14846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:57:34.915429 14846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:34.915589 14846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:57:34.915688 14846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:34.956019 14846 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40123
I20260927 06:57:34.956079 14993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40123 every 8 connection(s)
I20260927 06:57:34.957361 14846 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:57:34.958308 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 14846
I20260927 06:57:34.958999 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44123
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:36617
--webserver_port=36797
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 06:57:34 dist-test-slave-f892 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:34.981042 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:34.961450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42829 (local address 127.0.97.62:36617)
0927 06:57:34.962711 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:34.962728 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:34.962778 (+ 50us) server_negotiation.cc:407] Connection header received
0927 06:57:34.964295 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:34.964307 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:34.964615 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:34.964987 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:34.966534 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:34.967769 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:34.969193 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:34.969798 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:34.969999 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:34.973678 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:34.973691 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:34.973710 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:34.973765 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:34.976875 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:34.977567 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:34.977573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:34.977578 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:34.977722 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:34.978204 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:34.978208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:34.978210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:34.978646 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:34.978860 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:34.979589 (+ 729us) server_negotiation.cc:299] Negotiation successful
0927 06:57:34.979946 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1046,"thread_start_us":908,"threads_started":1}
I20260927 06:57:34.981040 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:34.961021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:42829)
0927 06:57:34.961856 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:34.961898 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:34.963764 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:34.965147 (+ 1383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:34.965160 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:34.965752 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:34.966314 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:34.966331 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:34.967934 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:34.967938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:34.969054 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:34.969067 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:34.970057 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:34.970065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:34.970332 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:34.971029 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:34.971055 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:34.973382 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:34.977020 (+ 3638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:34.977029 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:34.977041 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:34.977424 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:34.977832 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:34.977838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:34.977841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:34.978077 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:34.978794 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:34.978800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:34.979142 (+ 342us) client_negotiation.cc:769] Sending connection context
0927 06:57:34.979383 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 06:57:34.979840 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":347,"threads_started":1}
I20260927 06:57:34.983785 14994 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36617
I20260927 06:57:34.984088 14994 heartbeater.cc:464] Registering TS with master...
I20260927 06:57:34.984905 14994 heartbeater.cc:511] Master 127.0.97.62:36617 requested a full tablet report, sending...
I20260927 06:57:34.987123 14809 ts_manager.cc:194] Registered new tserver with Master: 01c30625ed884c11959092b7148a8239 (127.0.97.1:40123)
I20260927 06:57:34.989037 14809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42829
W20260927 06:57:35.272799 14998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:35.273049 14998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:35.273140 14998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:35.280131 14998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:35.280223 14998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:35.280328 14998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:57:35.289500 14998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36797
--tserver_master_addrs=127.0.97.62:36617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:35.290992 14998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:35.292243 14998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:35.304391 15006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:35.304359 15004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:35.304826 15003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:35.305215 14998 server_base.cc:1034] running on GCE node
I20260927 06:57:35.305872 14998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:35.306757 14998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:35.307971 14998 hybrid_clock.cc:648] HybridClock initialized: now 1790492255307926 us; error 55 us; skew 500 ppm
I20260927 06:57:35.308303 14998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:35 dist-test-slave-f892 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:35.311228 14998 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:57:35.312512 14998 webserver.cc:460] Webserver started at http://127.0.97.2:36797/ using document root <none> and password file <none>
I20260927 06:57:35.313135 14998 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:35.313300 14998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:35.318265 14998 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:57:35.320735 15013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:35.321985 14998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 06:57:35.322204 14998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "8931a6ee797b43d291c219d5f0b1404e"
format_stamp: "Formatted at 2026-09-27 06:57:32 on dist-test-slave-f892"
server_key: "b70d291a77ba8e941028dcf46c0c3a88"
server_key_iv: "0fd8399e10c377b9e50c9214d2abda32"
server_key_version: "encryptionkey@0"
I20260927 06:57:35.323135 14998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:35.339061 14998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:35.343092 14998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:35.343351 14998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:35.344262 14998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:57:35.345489 14998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:57:35.345629 14998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:35.345762 14998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:57:35.345845 14998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:35.377187 14998 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44123
I20260927 06:57:35.377267 15125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44123 every 8 connection(s)
I20260927 06:57:35.378649 14998 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:57:35.381088 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 14998
I20260927 06:57:35.381680 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44137
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:36617
--webserver_port=36713
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 06:57:35 dist-test-slave-f892 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:35.400697 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:35.381083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34363 (local address 127.0.97.62:36617)
0927 06:57:35.381315 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:35.381321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:35.382640 (+ 1319us) server_negotiation.cc:407] Connection header received
0927 06:57:35.384566 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:35.384574 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:35.384658 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:35.384722 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:35.386603 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.387297 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.388861 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.389265 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.389396 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:35.393499 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:35.393508 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:35.393516 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:35.393560 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:35.397236 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:35.397926 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:35.397935 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:35.397949 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:35.398153 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:35.398596 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:35.398604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:35.398610 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:35.399020 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:35.399180 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:35.399933 (+ 753us) server_negotiation.cc:299] Negotiation successful
0927 06:57:35.400219 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 06:57:35.401242 15128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:35.381513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:34363)
0927 06:57:35.382501 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:35.382543 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:35.384287 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:35.385173 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:35.385184 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:35.385814 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:35.386421 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:35.386437 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.387459 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.387468 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:35.388727 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:35.388738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.389472 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.389479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:35.389805 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:35.390530 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:35.390562 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:35.393280 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:35.397368 (+ 4088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:35.397374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:35.397386 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:35.397786 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:35.398273 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:35.398280 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:35.398283 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:35.398461 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:35.399145 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:35.399151 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:35.399521 (+ 370us) client_negotiation.cc:769] Sending connection context
0927 06:57:35.399782 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 06:57:35.400174 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":566,"threads_started":1}
I20260927 06:57:35.402714 15126 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36617
I20260927 06:57:35.403056 15126 heartbeater.cc:464] Registering TS with master...
I20260927 06:57:35.403999 15126 heartbeater.cc:511] Master 127.0.97.62:36617 requested a full tablet report, sending...
I20260927 06:57:35.405584 14809 ts_manager.cc:194] Registered new tserver with Master: 8931a6ee797b43d291c219d5f0b1404e (127.0.97.2:44123)
I20260927 06:57:35.406455 14809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34363
W20260927 06:57:35.701697 15129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:35.701924 15129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:35.702108 15129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:35.709175 15129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:35.709285 15129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:35.709460 15129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:57:35.718741 15129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36713
--tserver_master_addrs=127.0.97.62:36617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:35.719854 15129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:35.721016 15129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:35.733708 15135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:35.733774 15134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:35.733708 15137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:35.739279 15129 server_base.cc:1034] running on GCE node
I20260927 06:57:35.739814 15129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:35.740698 15129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:35.741890 15129 hybrid_clock.cc:648] HybridClock initialized: now 1790492255741863 us; error 42 us; skew 500 ppm
I20260927 06:57:35.742184 15129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:35 dist-test-slave-f892 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:35.745069 15129 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:57:35.746551 15129 webserver.cc:460] Webserver started at http://127.0.97.3:36713/ using document root <none> and password file <none>
I20260927 06:57:35.747175 15129 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:35.747320 15129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:35.752305 15129 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 06:57:35.754799 15144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:35.756028 15129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:57:35.756196 15129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "454b008c6926463e9778ee64e8c06a84"
format_stamp: "Formatted at 2026-09-27 06:57:32 on dist-test-slave-f892"
server_key: "a73cdd61065142369eb0b2b5b6ca6503"
server_key_iv: "f1b2aeb534d38ed749f3b67bacf9fd88"
server_key_version: "encryptionkey@0"
I20260927 06:57:35.757067 15129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:35.790670 15129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:35.794521 15129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:35.794795 15129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:35.795738 15129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:57:35.796967 15129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:57:35.797086 15129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:35.797226 15129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:57:35.797314 15129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:35.829301 15129 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44137
I20260927 06:57:35.829376 15256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44137 every 8 connection(s)
I20260927 06:57:35.830648 15129 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:57:35.836321 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 15129
Sep 27 06:57:35 dist-test-slave-f892 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:35.850314 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:35.832989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55953 (local address 127.0.97.62:36617)
0927 06:57:35.833179 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:35.833184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:35.834290 (+ 1106us) server_negotiation.cc:407] Connection header received
0927 06:57:35.836049 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:35.836054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:35.836148 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:35.836265 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:35.837901 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.838747 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.840191 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.840712 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.840835 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:35.844396 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:35.844401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:35.844404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:35.844434 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:35.847501 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:35.848174 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:35.848180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:35.848183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:35.848313 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:35.848698 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:35.848704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:35.848708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:35.848955 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:35.849107 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:35.849750 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 06:57:35.849953 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 06:57:35.850870 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:35.837130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55440 (local address 127.0.97.62:36617)
0927 06:57:35.837616 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:35.837621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:35.837991 (+ 370us) server_negotiation.cc:407] Connection header received
0927 06:57:35.838209 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:35.838214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:35.838285 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:35.838394 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:35.839810 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.840626 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.842058 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.842476 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.842605 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:35.845547 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:35.845559 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:35.845565 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:35.845603 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:35.848174 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:35.848811 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:35.848816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:35.848820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:35.848951 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:35.849312 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:35.849317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:35.849320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:35.849573 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:35.849891 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:35.850191 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 06:57:35.850404 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":259,"threads_started":1}
I20260927 06:57:35.851321 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:35.833421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:55953)
0927 06:57:35.834146 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:35.834181 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:35.835782 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:35.836521 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:35.836530 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:35.837050 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:35.837736 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:35.837755 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.838901 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.838913 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:35.840048 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:35.840059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.840927 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.840936 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:35.841162 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:35.841822 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:35.841845 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:35.844195 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:35.847671 (+ 3476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:35.847678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:35.847691 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:35.848047 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:35.848419 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:35.848424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:35.848427 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:35.848592 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:35.849082 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:35.849089 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:35.849362 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 06:57:35.849620 (+ 258us) client_negotiation.cc:240] Negotiation successful
0927 06:57:35.850125 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":272,"threads_started":1}
I20260927 06:57:35.852962 15257 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36617
I20260927 06:57:35.853286 15257 heartbeater.cc:464] Registering TS with master...
I20260927 06:57:35.853920 15257 heartbeater.cc:511] Master 127.0.97.62:36617 requested a full tablet report, sending...
I20260927 06:57:35.855129 14809 ts_manager.cc:194] Registered new tserver with Master: 454b008c6926463e9778ee64e8c06a84 (127.0.97.3:44137)
I20260927 06:57:35.855873 14809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55953
I20260927 06:57:35.865020 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:57:35.871178 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:35.865781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.0.97.62:36617)
0927 06:57:35.865943 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:35.865949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:35.866368 (+ 419us) server_negotiation.cc:407] Connection header received
0927 06:57:35.866592 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:35.866596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:35.866709 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:35.866842 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:57:35.867214 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.867844 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.869227 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.869615 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.869740 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:35.870237 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:57:35.870646 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:57:35.870795 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:35.870871 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 06:57:35.870940 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 06:57:35.871928 14809 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:57:35.878824 14809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:57:35.899160 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:35.889873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.0.97.1:40123)
0927 06:57:35.891656 (+ 1783us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:35.891666 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:35.891694 (+ 28us) server_negotiation.cc:407] Connection header received
0927 06:57:35.891794 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:35.891804 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:35.892003 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:35.892196 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:35.893037 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.894604 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.896702 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.898068 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.898220 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:35.898407 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:35.898676 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 06:57:35.898828 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1613,"thread_start_us":1650,"threads_started":1}
I20260927 06:57:35.899204 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:35.889363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40123 (local address 127.0.0.1:55748)
0927 06:57:35.890253 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:35.890295 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:35.890429 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:35.892345 (+ 1916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:35.892351 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:35.892382 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:35.892748 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:35.892863 (+ 115us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.894773 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.894778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:35.896537 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:35.896551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.898265 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.898271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:35.898443 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:35.898492 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 06:57:35.898616 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 06:57:35.898798 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":3,"thread_start_us":565,"threads_started":1}
I20260927 06:57:35.899870 15268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:35.889410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44137 (local address 127.0.0.1:52036)
0927 06:57:35.891147 (+ 1737us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:35.891166 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:35.891268 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:35.892020 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:35.892027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:35.892066 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:35.892525 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:35.892541 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.894707 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.894712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:35.896958 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:35.896992 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.899017 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.899023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:35.899198 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:35.899230 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 06:57:35.899319 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 06:57:35.899449 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1472,"mutex_wait_us":26,"thread_start_us":1355,"threads_started":1}
I20260927 06:57:35.900578 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:35.889484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52036 (local address 127.0.97.3:44137)
0927 06:57:35.891387 (+ 1903us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:35.891396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:35.891440 (+ 44us) server_negotiation.cc:407] Connection header received
0927 06:57:35.891529 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:35.891536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:35.891780 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:35.892747 (+ 967us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:35.892994 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.894572 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.897152 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.898539 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.898659 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:35.898857 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:35.900076 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0927 06:57:35.900237 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1714,"thread_start_us":1577,"threads_started":1}
I20260927 06:57:35.900650 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:35.891191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44123 (local address 127.0.0.1:40480)
0927 06:57:35.892068 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:35.892083 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:35.892209 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:35.893264 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:35.893269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:35.893291 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:35.893648 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:35.893659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.895495 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.895503 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:35.897738 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:35.897751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.899847 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.899859 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:35.900066 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:35.900098 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 06:57:35.900189 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 06:57:35.900294 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":24,"thread_start_us":885,"threads_started":1}
I20260927 06:57:35.901412 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:35.891829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40480 (local address 127.0.97.2:44123)
0927 06:57:35.892809 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:35.892815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:35.892838 (+ 23us) server_negotiation.cc:407] Connection header received
0927 06:57:35.892908 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:35.892912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:35.893105 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:35.893284 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:35.893835 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.895353 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.897909 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:35.899361 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:35.899462 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:35.899681 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:35.900250 (+ 569us) server_negotiation.cc:299] Negotiation successful
0927 06:57:35.900400 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":719,"threads_started":1}
I20260927 06:57:35.903522 15192 tablet_service.cc:1467] Processing CreateTablet for tablet 9c1cd5e2e1c241de9115e0e82232d62b (DEFAULT_TABLE table=test-table [id=343039508ada4209859f9bbd65e75a34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:57:35.903410 14929 tablet_service.cc:1467] Processing CreateTablet for tablet 9c1cd5e2e1c241de9115e0e82232d62b (DEFAULT_TABLE table=test-table [id=343039508ada4209859f9bbd65e75a34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:57:35.904621 15192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c1cd5e2e1c241de9115e0e82232d62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:35.904700 14929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c1cd5e2e1c241de9115e0e82232d62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:35.904903 15061 tablet_service.cc:1467] Processing CreateTablet for tablet 9c1cd5e2e1c241de9115e0e82232d62b (DEFAULT_TABLE table=test-table [id=343039508ada4209859f9bbd65e75a34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:57:35.906548 15061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c1cd5e2e1c241de9115e0e82232d62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:35.917784 15273 tablet_bootstrap.cc:492] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84: Bootstrap starting.
I20260927 06:57:35.921066 15273 tablet_bootstrap.cc:654] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84: Neither blocks nor log segments found. Creating new log.
I20260927 06:57:35.921532 15275 tablet_bootstrap.cc:492] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e: Bootstrap starting.
I20260927 06:57:35.922411 15273 log.cc:824] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:35.924875 15275 tablet_bootstrap.cc:654] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e: Neither blocks nor log segments found. Creating new log.
I20260927 06:57:35.925473 15273 tablet_bootstrap.cc:492] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84: No bootstrap required, opened a new log
I20260927 06:57:35.925740 15273 ts_tablet_manager.cc:1397] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 06:57:35.926059 15275 log.cc:824] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:35.926100 15274 tablet_bootstrap.cc:492] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239: Bootstrap starting.
I20260927 06:57:35.929365 15274 tablet_bootstrap.cc:654] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239: Neither blocks nor log segments found. Creating new log.
I20260927 06:57:35.929936 15275 tablet_bootstrap.cc:492] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e: No bootstrap required, opened a new log
I20260927 06:57:35.930194 15275 ts_tablet_manager.cc:1397] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 06:57:35.930325 15274 log.cc:824] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:35.932509 15274 tablet_bootstrap.cc:492] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239: No bootstrap required, opened a new log
I20260927 06:57:35.932758 15274 ts_tablet_manager.cc:1397] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20260927 06:57:35.935895 15273 raft_consensus.cc:348] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } } peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } }
I20260927 06:57:35.936302 15273 raft_consensus.cc:374] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:35.936398 15273 raft_consensus.cc:729] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 454b008c6926463e9778ee64e8c06a84, State: Initialized, Role: FOLLOWER
I20260927 06:57:35.936972 15273 consensus_queue.cc:260] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } } peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } }
I20260927 06:57:35.938216 15257 heartbeater.cc:503] Master 127.0.97.62:36617 was elected leader, sending a full tablet report...
I20260927 06:57:35.938719 15273 ts_tablet_manager.cc:1428] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260927 06:57:35.940346 15275 raft_consensus.cc:348] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } } peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } }
I20260927 06:57:35.940778 15275 raft_consensus.cc:374] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:35.940881 15275 raft_consensus.cc:729] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8931a6ee797b43d291c219d5f0b1404e, State: Initialized, Role: FOLLOWER
I20260927 06:57:35.941452 15275 consensus_queue.cc:260] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } } peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } }
I20260927 06:57:35.941938 15274 raft_consensus.cc:348] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } } peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } }
I20260927 06:57:35.942374 15274 raft_consensus.cc:374] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:35.942478 15274 raft_consensus.cc:729] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c30625ed884c11959092b7148a8239, State: Initialized, Role: FOLLOWER
I20260927 06:57:35.942600 15126 heartbeater.cc:503] Master 127.0.97.62:36617 was elected leader, sending a full tablet report...
I20260927 06:57:35.943109 15274 consensus_queue.cc:260] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } } peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } }
I20260927 06:57:35.944283 14994 heartbeater.cc:503] Master 127.0.97.62:36617 was elected leader, sending a full tablet report...
I20260927 06:57:35.944761 15274 ts_tablet_manager.cc:1428] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260927 06:57:35.945749 15275 ts_tablet_manager.cc:1428] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e: Time spent starting tablet: real 0.015s user 0.007s sys 0.005s
W20260927 06:57:35.961844 14995 tablet.cc:2369] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:57:36.003895 15280 raft_consensus.cc:482] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:57:36.004216 15280 raft_consensus.cc:504] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } } peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } }
I20260927 06:57:36.005925 15280 leader_election.cc:288] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 454b008c6926463e9778ee64e8c06a84 (127.0.97.3:44137), 01c30625ed884c11959092b7148a8239 (127.0.97.1:40123)
I20260927 06:57:36.013445 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:36.006413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58891 (local address 127.0.97.1:40123)
0927 06:57:36.006718 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:36.006726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:36.007607 (+ 881us) server_negotiation.cc:407] Connection header received
0927 06:57:36.007726 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:36.007731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:36.007800 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:36.007894 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:36.008623 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:36.009263 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:36.011159 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:36.012379 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:36.012526 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:36.012647 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:36.013066 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 06:57:36.013192 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260927 06:57:36.013445 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:36.006491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41785 (local address 127.0.97.3:44137)
0927 06:57:36.006680 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:36.006685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:36.007495 (+ 810us) server_negotiation.cc:407] Connection header received
0927 06:57:36.007722 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:36.007727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:36.007800 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:36.007893 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:36.008446 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:36.009218 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:36.011332 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:36.012515 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:36.012623 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:36.012721 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:36.013066 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 06:57:36.013196 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 06:57:36.013685 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:36.006391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44137 (local address 127.0.97.2:41785)
0927 06:57:36.007370 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:36.007394 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:36.007558 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:36.007925 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:36.007932 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:36.007959 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:36.008290 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:36.008298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:36.009351 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:36.009357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:36.011178 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:36.011193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:36.012711 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:36.012716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:36.012876 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:36.012927 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 06:57:36.013028 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 06:57:36.013222 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":126,"thread_start_us":540,"threads_started":1}
I20260927 06:57:36.013971 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:36.006472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40123 (local address 127.0.97.2:58891)
0927 06:57:36.007513 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:36.007528 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:36.007629 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:36.008073 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:36.008084 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:36.008108 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:36.008465 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:36.008476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:36.009424 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:36.009428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:36.011015 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:36.011026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:36.012672 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:36.012680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:36.012871 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:36.012940 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 06:57:36.013039 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 06:57:36.013222 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":85,"thread_start_us":555,"threads_started":1}
I20260927 06:57:36.014670 15212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1cd5e2e1c241de9115e0e82232d62b" candidate_uuid: "8931a6ee797b43d291c219d5f0b1404e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "454b008c6926463e9778ee64e8c06a84" is_pre_election: true
I20260927 06:57:36.014890 14949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1cd5e2e1c241de9115e0e82232d62b" candidate_uuid: "8931a6ee797b43d291c219d5f0b1404e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c30625ed884c11959092b7148a8239" is_pre_election: true
I20260927 06:57:36.015110 15212 raft_consensus.cc:2454] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8931a6ee797b43d291c219d5f0b1404e in term 0.
I20260927 06:57:36.015420 14949 raft_consensus.cc:2454] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8931a6ee797b43d291c219d5f0b1404e in term 0.
I20260927 06:57:36.015795 15015 leader_election.cc:302] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 454b008c6926463e9778ee64e8c06a84, 8931a6ee797b43d291c219d5f0b1404e; no voters:
I20260927 06:57:36.016201 15280 raft_consensus.cc:2781] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:57:36.016347 15280 raft_consensus.cc:482] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:57:36.016465 15280 raft_consensus.cc:3037] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:36.018710 15280 raft_consensus.cc:504] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } } peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } }
I20260927 06:57:36.019193 15280 leader_election.cc:288] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [CANDIDATE]: Term 1 election: Requested vote from peers 454b008c6926463e9778ee64e8c06a84 (127.0.97.3:44137), 01c30625ed884c11959092b7148a8239 (127.0.97.1:40123)
I20260927 06:57:36.019580 15212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1cd5e2e1c241de9115e0e82232d62b" candidate_uuid: "8931a6ee797b43d291c219d5f0b1404e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "454b008c6926463e9778ee64e8c06a84"
I20260927 06:57:36.019577 14949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1cd5e2e1c241de9115e0e82232d62b" candidate_uuid: "8931a6ee797b43d291c219d5f0b1404e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c30625ed884c11959092b7148a8239"
I20260927 06:57:36.019752 15212 raft_consensus.cc:3037] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:36.019737 14949 raft_consensus.cc:3037] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:36.021611 14949 raft_consensus.cc:2454] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8931a6ee797b43d291c219d5f0b1404e in term 1.
I20260927 06:57:36.021625 15212 raft_consensus.cc:2454] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8931a6ee797b43d291c219d5f0b1404e in term 1.
I20260927 06:57:36.021979 15016 leader_election.cc:302] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c30625ed884c11959092b7148a8239, 8931a6ee797b43d291c219d5f0b1404e; no voters:
I20260927 06:57:36.022286 15280 raft_consensus.cc:2781] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:57:36.022693 15280 raft_consensus.cc:686] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [term 1 LEADER]: Becoming Leader. State: Replica: 8931a6ee797b43d291c219d5f0b1404e, State: Running, Role: LEADER
I20260927 06:57:36.023119 15280 consensus_queue.cc:237] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } } peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } }
I20260927 06:57:36.027535 14808 catalog_manager.cc:5495] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8931a6ee797b43d291c219d5f0b1404e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8931a6ee797b43d291c219d5f0b1404e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } health_report { overall_health: HEALTHY } } }
I20260927 06:57:36.052239 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:36.047686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40496 (local address 127.0.97.2:44123)
0927 06:57:36.047862 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:36.047866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:36.047881 (+ 15us) server_negotiation.cc:407] Connection header received
0927 06:57:36.048018 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:36.048022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:36.048092 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:36.048203 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:57:36.048510 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:36.049134 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:36.050448 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:36.050827 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:36.050911 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:36.051346 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:57:36.051800 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:57:36.051901 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:36.052000 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 06:57:36.052057 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 06:57:36.059451 15212 raft_consensus.cc:1261] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84 [term 1 FOLLOWER]: Refusing update from remote peer 8931a6ee797b43d291c219d5f0b1404e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:57:36.059504 14949 raft_consensus.cc:1261] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Refusing update from remote peer 8931a6ee797b43d291c219d5f0b1404e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:57:36.060424 15284 consensus_queue.cc:1035] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [LEADER]: Connected to new peer: Peer: permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:57:36.060765 15280 consensus_queue.cc:1035] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [LEADER]: Connected to new peer: Peer: permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:57:36.071151 15287 mvcc.cc:205] Tried to move back new op lower bound from 7333856280810496000 to 7333856280674189312. Current Snapshot: MvccSnapshot[applied={T|T < 7333856280810496000}]
I20260927 06:57:36.076258 15289 mvcc.cc:205] Tried to move back new op lower bound from 7333856280810496000 to 7333856280674189312. Current Snapshot: MvccSnapshot[applied={T|T < 7333856280810496000}]
I20260927 06:57:36.084520 14809 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:57:36.084789 14809 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:57:36.088112 14809 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069: Sending DeleteTablet for 3 replicas of tablet 9c1cd5e2e1c241de9115e0e82232d62b
I20260927 06:57:36.089201 14929 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c1cd5e2e1c241de9115e0e82232d62b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:57:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55748
I20260927 06:57:36.089416 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 14846
I20260927 06:57:36.089449 15192 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c1cd5e2e1c241de9115e0e82232d62b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:57:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52036
I20260927 06:57:36.090139 15061 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c1cd5e2e1c241de9115e0e82232d62b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:57:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40480
I20260927 06:57:36.090698 15297 tablet_replica.cc:330] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84: stopping tablet replica
I20260927 06:57:36.091063 15297 raft_consensus.cc:2229] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:57:36.093304 15297 raft_consensus.cc:2258] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:57:36.113214 15297 ts_tablet_manager.cc:1905] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:57:36.119205 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:36.090794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.0.97.1:40123)
0927 06:57:36.090999 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:36.091007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:36.091029 (+ 22us) server_negotiation.cc:407] Connection header received
0927 06:57:36.091096 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:36.091104 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:36.091216 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:36.091340 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:36.093052 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:36.094584 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:36.096468 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:36.097358 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:36.097505 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:36.101981 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:36.102019 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:36.102041 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:36.102099 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:36.107349 (+ 5250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:36.108407 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:36.108420 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:36.108426 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:36.108614 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:36.109137 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:36.109145 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:36.109150 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:36.109730 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:36.110088 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:36.118375 (+ 8287us) server_negotiation.cc:299] Negotiation successful
0927 06:57:36.118687 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 06:57:36.119925 14989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:36.127223 15298 tablet_replica.cc:330] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e: stopping tablet replica
I20260927 06:57:36.128614 15298 raft_consensus.cc:2229] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:57:36.129428 15298 raft_consensus.cc:2258] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:57:36.131770 15298 ts_tablet_manager.cc:1905] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:57:36.138304 15297 ts_tablet_manager.cc:1918] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:57:36.138512 15297 log.cc:1196] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/9c1cd5e2e1c241de9115e0e82232d62b
I20260927 06:57:36.139199 15297 ts_tablet_manager.cc:1939] T 9c1cd5e2e1c241de9115e0e82232d62b P 454b008c6926463e9778ee64e8c06a84: Deleting consensus metadata
I20260927 06:57:36.139381 15298 ts_tablet_manager.cc:1918] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:57:36.139585 15298 log.cc:1196] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/9c1cd5e2e1c241de9115e0e82232d62b
I20260927 06:57:36.140146 15298 ts_tablet_manager.cc:1939] T 9c1cd5e2e1c241de9115e0e82232d62b P 8931a6ee797b43d291c219d5f0b1404e: Deleting consensus metadata
I20260927 06:57:36.141299 14794 catalog_manager.cc:4841] TS 454b008c6926463e9778ee64e8c06a84 (127.0.97.3:44137): tablet 9c1cd5e2e1c241de9115e0e82232d62b (table test-table [id=343039508ada4209859f9bbd65e75a34]) successfully deleted
I20260927 06:57:36.141866 14794 catalog_manager.cc:4841] TS 8931a6ee797b43d291c219d5f0b1404e (127.0.97.2:44123): tablet 9c1cd5e2e1c241de9115e0e82232d62b (table test-table [id=343039508ada4209859f9bbd65e75a34]) successfully deleted
I20260927 06:57:36.248093 15296 tablet_replica.cc:330] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239: stopping tablet replica
I20260927 06:57:36.248418 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 14846
I20260927 06:57:36.265812 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 14998
W20260927 06:57:36.265739 14795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40123 (error 108)
I20260927 06:57:36.267926 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:36.267357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40123 (local address 127.0.0.1:55774)
0927 06:57:36.267644 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:36.267770 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260927 06:57:36.268157 14795 catalog_manager.cc:4584] TS 01c30625ed884c11959092b7148a8239 (127.0.97.1:40123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c1cd5e2e1c241de9115e0e82232d62b: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40123: connect: Connection refused (error 111)
I20260927 06:57:36.285351 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:36.266632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40498 (local address 127.0.97.2:44123)
0927 06:57:36.266802 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:36.266808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:36.267058 (+ 250us) server_negotiation.cc:407] Connection header received
0927 06:57:36.267130 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:36.267135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:36.267215 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:36.267369 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:36.268876 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:36.270055 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:36.271230 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:36.271935 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:36.272069 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:36.278677 (+ 6608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:36.278694 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:36.278712 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:36.278751 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:36.282579 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:36.283274 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:36.283283 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:36.283287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:36.283449 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:36.283814 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:36.283819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:36.283823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:36.284253 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:36.284593 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:36.284738 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 06:57:36.284954 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 06:57:36.286163 15121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:36.321830 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:36.321277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40123 (local address 127.0.0.1:55790)
0927 06:57:36.321587 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:36.321682 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260927 06:57:36.322939 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:36.322582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40123 (local address 127.0.0.1:55796)
0927 06:57:36.322763 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:36.322839 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 06:57:36.383349 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:36.382878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40123 (local address 127.0.0.1:55804)
0927 06:57:36.383058 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:36.383142 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 06:57:36.384298 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:36.383967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40123 (local address 127.0.0.1:55820)
0927 06:57:36.384109 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:36.384200 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 06:57:36.397658 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 14998
I20260927 06:57:36.416157 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 15129
I20260927 06:57:36.429121 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:36.416823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52046 (local address 127.0.97.3:44137)
0927 06:57:36.416999 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:36.417005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:36.417025 (+ 20us) server_negotiation.cc:407] Connection header received
0927 06:57:36.417216 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:36.417225 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:36.417284 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:36.417380 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:36.418718 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:36.419389 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:36.420598 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:36.420989 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:36.421110 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:36.423803 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:36.423815 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:36.423829 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:36.423863 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:36.426442 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:36.427114 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:36.427121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:36.427123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:36.427256 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:36.427617 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:36.427622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:36.427626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:36.427946 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:36.428243 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:36.428600 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 06:57:36.428820 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 06:57:36.429702 15252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:36.464803 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:36.464321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40123 (local address 127.0.0.1:55830)
0927 06:57:36.464539 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:36.464638 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 06:57:36.465819 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:36.465409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40123 (local address 127.0.0.1:55838)
0927 06:57:36.465611 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:36.465696 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 06:57:36.516716 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 15129
I20260927 06:57:36.532656 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 14776
I20260927 06:57:36.533730 14839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:36.643409 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 14776
I20260927 06:57:36.664987 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36617
--webserver_interface=127.0.97.62
--webserver_port=41467
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:57:36.978893 15304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:36.979189 15304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:36.979354 15304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:36.986320 15304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:57:36.986456 15304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:36.986573 15304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:36.986670 15304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:57:36.986756 15304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:57:36.996004 15304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=41467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:36.997201 15304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:36.998523 15304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:37.009433 15310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:37.009522 15312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:37.009500 15309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:37.010504 15304 server_base.cc:1034] running on GCE node
I20260927 06:57:37.011241 15304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:37.012444 15304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:37.013641 15304 hybrid_clock.cc:648] HybridClock initialized: now 1790492257013578 us; error 45 us; skew 500 ppm
I20260927 06:57:37.014065 15304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:37 dist-test-slave-f892 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:37.021171 15304 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:57:37.022686 15304 webserver.cc:460] Webserver started at http://127.0.97.62:41467/ using document root <none> and password file <none>
I20260927 06:57:37.023362 15304 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:37.023492 15304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:37.028690 15304 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 06:57:37.031329 15319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:37.032550 15304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 06:57:37.032709 15304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "e64894159daa4058a93c6b59fe0d3069"
format_stamp: "Formatted at 2026-09-27 06:57:31 on dist-test-slave-f892"
server_key: "0c635fdc60d0de862586a0e39d353139"
server_key_iv: "46e513a5080467ed853ecb1fbf0da697"
server_key_version: "encryptionkey@0"
I20260927 06:57:37.033643 15304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:37.056727 15304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:37.060456 15304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:37.060721 15304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:37.078493 15304 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36617
I20260927 06:57:37.078559 15370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36617 every 8 connection(s)
I20260927 06:57:37.079885 15304 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:57:37.085565 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 15304
I20260927 06:57:37.086573 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40123
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:36617
--webserver_port=42303
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:57:37.087195 15371 sys_catalog.cc:263] Verifying existing consensus state
I20260927 06:57:37.092173 15371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069: Bootstrap starting.
I20260927 06:57:37.112054 15371 log.cc:824] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:37.127345 15371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 06:57:37.127856 15371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069: Bootstrap complete.
I20260927 06:57:37.136857 15371 raft_consensus.cc:348] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64894159daa4058a93c6b59fe0d3069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36617 } }
I20260927 06:57:37.138239 15371 raft_consensus.cc:729] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e64894159daa4058a93c6b59fe0d3069, State: Initialized, Role: FOLLOWER
I20260927 06:57:37.138885 15371 consensus_queue.cc:260] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64894159daa4058a93c6b59fe0d3069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36617 } }
I20260927 06:57:37.139106 15371 raft_consensus.cc:388] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:57:37.139221 15371 raft_consensus.cc:482] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:57:37.139398 15371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 2 FOLLOWER]: Advancing to term 3
I20260927 06:57:37.142885 15371 raft_consensus.cc:504] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64894159daa4058a93c6b59fe0d3069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36617 } }
I20260927 06:57:37.143360 15371 leader_election.cc:302] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e64894159daa4058a93c6b59fe0d3069; no voters:
I20260927 06:57:37.144001 15371 leader_election.cc:288] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 06:57:37.144182 15375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 06:57:37.145237 15375 raft_consensus.cc:686] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [term 3 LEADER]: Becoming Leader. State: Replica: e64894159daa4058a93c6b59fe0d3069, State: Running, Role: LEADER
I20260927 06:57:37.145920 15371 sys_catalog.cc:564] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:57:37.145795 15375 consensus_queue.cc:237] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64894159daa4058a93c6b59fe0d3069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36617 } }
I20260927 06:57:37.148016 15377 sys_catalog.cc:455] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e64894159daa4058a93c6b59fe0d3069. Latest consensus state: current_term: 3 leader_uuid: "e64894159daa4058a93c6b59fe0d3069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64894159daa4058a93c6b59fe0d3069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36617 } } }
I20260927 06:57:37.148397 15377 sys_catalog.cc:458] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [sys.catalog]: This master's current role is: LEADER
I20260927 06:57:37.148906 15376 sys_catalog.cc:455] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e64894159daa4058a93c6b59fe0d3069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64894159daa4058a93c6b59fe0d3069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36617 } } }
I20260927 06:57:37.149154 15376 sys_catalog.cc:458] T 00000000000000000000000000000000 P e64894159daa4058a93c6b59fe0d3069 [sys.catalog]: This master's current role is: LEADER
I20260927 06:57:37.153672 15381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:57:37.163196 15381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:57:37.165107 15381 catalog_manager.cc:1259] Loaded cluster ID: bff8cdfe56fe470ea385bef9b9b3f970
I20260927 06:57:37.165301 15381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 06:57:37.167900 15381 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe52bc739ad google::LogMessage::Fail() at ??:0
@ 0x7fe52bc7688a google::LogMessage::SendToLog() at ??:0
@ 0x7fe52bc734de google::LogMessage::Flush() at ??:0
@ 0x7fe52bc74a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe5416a6512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe5416a3976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe54172b610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
I20260927 06:57:37.200836 15396 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fe54172b439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe52e8bf345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe52e8c291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe52e8c2799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe52e89deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe531b776db start_thread at ??:0
@ 0x7fe52ab0d61f clone at ??:0
W20260927 06:57:37.440263 15373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:37.440593 15373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:37.440766 15373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:37.448001 15373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:37.448145 15373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:37.448311 15373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:57:37.457773 15373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42303
--tserver_master_addrs=127.0.97.62:36617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:37.459030 15373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:37.460281 15373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:37.472855 15413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:37.472836 15414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:37.472797 15416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:37.473806 15373 server_base.cc:1034] running on GCE node
I20260927 06:57:37.474431 15373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:37.475277 15373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:37.476464 15373 hybrid_clock.cc:648] HybridClock initialized: now 1790492257476430 us; error 27 us; skew 500 ppm
I20260927 06:57:37.476734 15373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:37 dist-test-slave-f892 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:37.479775 15373 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:57:37.480974 15373 webserver.cc:460] Webserver started at http://127.0.97.1:42303/ using document root <none> and password file <none>
I20260927 06:57:37.481570 15373 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:37.481673 15373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:37.486753 15373 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 06:57:37.489380 15423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:37.490620 15373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 06:57:37.490739 15373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "01c30625ed884c11959092b7148a8239"
format_stamp: "Formatted at 2026-09-27 06:57:32 on dist-test-slave-f892"
server_key: "fd770bcfb03fbfcd13740ed33ca7b0e9"
server_key_iv: "d1a1e42f4e17c6694cf196d2aa6c4397"
server_key_version: "encryptionkey@0"
I20260927 06:57:37.491605 15373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:37.528384 15373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:37.532526 15373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:37.532740 15373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:37.533656 15373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:57:37.535324 15430 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260927 06:57:37.540531 15373 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 06:57:37.540668 15373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260927 06:57:37.540807 15373 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 06:57:37.543751 15373 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 06:57:37.543824 15373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260927 06:57:37.544198 15430 tablet_bootstrap.cc:492] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239: Bootstrap starting.
I20260927 06:57:37.571283 15430 log.cc:824] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:37.581715 15430 tablet_bootstrap.cc:492] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 06:57:37.582521 15430 tablet_bootstrap.cc:492] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239: Bootstrap complete.
I20260927 06:57:37.584342 15430 ts_tablet_manager.cc:1397] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239: Time spent bootstrapping tablet: real 0.040s user 0.026s sys 0.012s
I20260927 06:57:37.587952 15373 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40123
I20260927 06:57:37.588070 15537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40123 every 8 connection(s)
I20260927 06:57:37.589188 15373 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:57:37.590631 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 15373
I20260927 06:57:37.591566 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44123
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:36617
--webserver_port=36797
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:57:37.594179 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:37.591872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:41357)
0927 06:57:37.592717 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:37.592955 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":375,"threads_started":1}
W20260927 06:57:37.594758 15538 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36617 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:36617: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
I20260927 06:57:37.595886 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:37.595314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:37547)
0927 06:57:37.595736 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:37.595796 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333}
I20260927 06:57:37.596942 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:37.596520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:39719)
0927 06:57:37.596691 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:37.596796 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 06:57:37.597188 15538 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 06:57:37.597040 15430 raft_consensus.cc:348] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } } peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } }
I20260927 06:57:37.597982 15430 raft_consensus.cc:729] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c30625ed884c11959092b7148a8239, State: Initialized, Role: FOLLOWER
I20260927 06:57:37.598621 15430 consensus_queue.cc:260] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } } peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } }
I20260927 06:57:37.600186 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:37.599842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:47873)
0927 06:57:37.600005 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:37.600066 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 06:57:37.600203 15430 ts_tablet_manager.cc:1428] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260927 06:57:37.601147 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:37.600862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:56679)
0927 06:57:37.600990 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:37.601047 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 06:57:37.907094 15542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:37.907358 15542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:37.907450 15542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:37.914629 15542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:37.914805 15542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:37.915020 15542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:57:37.925493 15542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36797
--tserver_master_addrs=127.0.97.62:36617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:37.926855 15542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:37.928146 15542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:37.941262 15550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:37.941349 15548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:37.941318 15547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:37.942253 15542 server_base.cc:1034] running on GCE node
I20260927 06:57:37.942824 15542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:37.943660 15542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:37.944883 15542 hybrid_clock.cc:648] HybridClock initialized: now 1790492257944834 us; error 35 us; skew 500 ppm
I20260927 06:57:37.945271 15542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:37 dist-test-slave-f892 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:37.948475 15542 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:57:37.949855 15542 webserver.cc:460] Webserver started at http://127.0.97.2:36797/ using document root <none> and password file <none>
I20260927 06:57:37.950531 15542 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:37.950636 15542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:37.955714 15542 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 06:57:37.960070 15557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:37.961272 15542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 06:57:37.961440 15542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "8931a6ee797b43d291c219d5f0b1404e"
format_stamp: "Formatted at 2026-09-27 06:57:32 on dist-test-slave-f892"
server_key: "b70d291a77ba8e941028dcf46c0c3a88"
server_key_iv: "0fd8399e10c377b9e50c9214d2abda32"
server_key_version: "encryptionkey@0"
I20260927 06:57:37.962415 15542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:37.982486 15542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:37.986305 15542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:37.986557 15542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:37.987430 15542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:57:37.988644 15542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:57:37.988794 15542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:37.988937 15542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:57:37.989022 15542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:38.021757 15542 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44123
I20260927 06:57:38.021842 15669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44123 every 8 connection(s)
I20260927 06:57:38.023233 15542 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:57:38.025381 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 15542
I20260927 06:57:38.025956 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44137
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:36617
--webserver_port=36713
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:57:38.028296 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:38.026121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:59527)
0927 06:57:38.027005 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:38.027253 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":280,"threads_started":1}
W20260927 06:57:38.028913 15670 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36617 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:36617: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
I20260927 06:57:38.029842 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:38.029459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:47685)
0927 06:57:38.029636 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:38.029712 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 06:57:38.030781 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:38.030450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:58839)
0927 06:57:38.030600 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:38.030672 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 06:57:38.031121 15670 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 06:57:38.335053 15673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:38.335306 15673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:38.335392 15673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:38.342370 15673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:38.342463 15673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:38.342644 15673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:57:38.352381 15673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36713
--tserver_master_addrs=127.0.97.62:36617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:38.353550 15673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:38.354768 15673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:38.367400 15681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:38.367441 15679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:38.367468 15678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:38.368156 15673 server_base.cc:1034] running on GCE node
I20260927 06:57:38.368770 15673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:38.369673 15673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:38.370884 15673 hybrid_clock.cc:648] HybridClock initialized: now 1790492258370848 us; error 50 us; skew 500 ppm
I20260927 06:57:38.371189 15673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:38 dist-test-slave-f892 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:38.374181 15673 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:57:38.375490 15673 webserver.cc:460] Webserver started at http://127.0.97.3:36713/ using document root <none> and password file <none>
I20260927 06:57:38.376139 15673 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:38.376282 15673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:38.381208 15673 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 06:57:38.383857 15688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:38.385059 15673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 06:57:38.385238 15673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "454b008c6926463e9778ee64e8c06a84"
format_stamp: "Formatted at 2026-09-27 06:57:32 on dist-test-slave-f892"
server_key: "a73cdd61065142369eb0b2b5b6ca6503"
server_key_iv: "f1b2aeb534d38ed749f3b67bacf9fd88"
server_key_version: "encryptionkey@0"
I20260927 06:57:38.386219 15673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:38.406575 15673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:38.410581 15673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:38.410948 15673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:38.411860 15673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:57:38.413100 15673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:57:38.413232 15673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:38.413383 15673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:57:38.413470 15673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:38.445549 15673 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44137
I20260927 06:57:38.445623 15800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44137 every 8 connection(s)
I20260927 06:57:38.446992 15673 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:57:38.447757 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 15673
I20260927 06:57:38.451586 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:38.449898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:39749)
0927 06:57:38.450606 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:38.450802 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":267,"threads_started":1}
W20260927 06:57:38.452157 15801 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36617 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:36617: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
I20260927 06:57:38.452901 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:38.452578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:45581)
0927 06:57:38.452736 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:38.452809 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 06:57:38.453672 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:38.453404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:45805)
0927 06:57:38.453511 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:38.453574 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 06:57:38.453907 15801 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 06:57:38.602659 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:38.601870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:56743)
0927 06:57:38.602378 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:38.602474 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":294,"threads_started":1}
I20260927 06:57:39.032337 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:39.031625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:54339)
0927 06:57:39.032032 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:39.032114 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":198,"threads_started":1}
I20260927 06:57:39.133350 15807 raft_consensus.cc:482] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:57:39.133664 15807 raft_consensus.cc:504] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } } peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } }
I20260927 06:57:39.135186 15807 leader_election.cc:288] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 454b008c6926463e9778ee64e8c06a84 (127.0.97.3:44137), 8931a6ee797b43d291c219d5f0b1404e (127.0.97.2:44123)
Sep 27 06:57:39 dist-test-slave-f892 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Sep 27 06:57:39 dist-test-slave-f892 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:57:39.156204 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:39.136195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44123 (local address 127.0.97.1:41655)
0927 06:57:39.137576 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:39.137601 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:39.137718 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:39.140201 (+ 2483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:39.140205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:39.140856 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:39.141520 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:39.141539 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:39.143069 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:39.143072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:39.144689 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:39.144701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:39.145625 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:39.145631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:39.145923 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:39.146756 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:39.146771 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:39.149533 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:39.152953 (+ 3420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:39.152956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:39.152958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:39.153539 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:39.153981 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:39.153986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:39.153991 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:39.154200 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:39.155179 (+ 979us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:39.155184 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:39.155323 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 06:57:39.155557 (+ 234us) client_negotiation.cc:240] Negotiation successful
0927 06:57:39.155911 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1245,"mutex_wait_us":139,"spinlock_wait_cycles":10624,"thread_start_us":1081,"threads_started":1}
I20260927 06:57:39.156584 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:39.136700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41655 (local address 127.0.97.2:44123)
0927 06:57:39.138237 (+ 1537us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:39.138246 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:39.138278 (+ 32us) server_negotiation.cc:407] Connection header received
0927 06:57:39.139666 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:39.139674 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:39.139954 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:39.140238 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:39.141701 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:39.142897 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:39.144828 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:39.145417 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:39.145616 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:39.149783 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:39.149792 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:39.149806 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:39.149848 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:39.152824 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:39.153686 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:39.153696 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:39.153703 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:39.153852 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:39.154307 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:39.154315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:39.154320 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:39.154939 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:39.155101 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:39.155797 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 06:57:39.156183 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1397,"thread_start_us":1210,"threads_started":1}
I20260927 06:57:39.156600 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:39.135671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45401 (local address 127.0.97.3:44137)
0927 06:57:39.137850 (+ 2179us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:39.137861 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:39.137892 (+ 31us) server_negotiation.cc:407] Connection header received
0927 06:57:39.139309 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:39.139317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:39.139648 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:39.139952 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:39.141701 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:39.142889 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:39.144829 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:39.145436 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:39.145616 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:39.149449 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:39.149455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:39.149467 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:39.149498 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:39.152746 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:39.153717 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:39.153722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:39.153727 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:39.153871 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:39.154349 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:39.154352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:39.154354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:39.154787 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:39.154971 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:39.155807 (+ 836us) server_negotiation.cc:299] Negotiation successful
0927 06:57:39.156183 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1989,"thread_start_us":1896,"threads_started":1}
I20260927 06:57:39.157497 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:39.135491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44137 (local address 127.0.97.1:45401)
0927 06:57:39.136064 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:39.136102 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:39.136753 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:39.140066 (+ 3313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:39.140077 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:39.140846 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:39.141520 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:39.141538 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:39.143076 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:39.143080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:39.144696 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:39.144707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:39.145662 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:39.145669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:39.145916 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:39.146696 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:39.146720 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:39.149187 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:39.152886 (+ 3699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:39.152897 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:39.152911 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:39.153554 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:39.154060 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:39.154065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:39.154069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:39.154227 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:39.154932 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:39.154939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:39.155269 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 06:57:39.156880 (+ 1611us) client_negotiation.cc:240] Negotiation successful
0927 06:57:39.157152 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":146,"thread_start_us":281,"threads_started":1}
W20260927 06:57:39.159564 15425 leader_election.cc:341] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8931a6ee797b43d291c219d5f0b1404e (127.0.97.2:44123): Not found: Tablet not found: 9c1cd5e2e1c241de9115e0e82232d62b
W20260927 06:57:39.159863 15425 leader_election.cc:341] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 454b008c6926463e9778ee64e8c06a84 (127.0.97.3:44137): Not found: Tablet not found: 9c1cd5e2e1c241de9115e0e82232d62b
I20260927 06:57:39.160010 15425 leader_election.cc:302] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01c30625ed884c11959092b7148a8239; no voters: 454b008c6926463e9778ee64e8c06a84, 8931a6ee797b43d291c219d5f0b1404e
I20260927 06:57:39.160372 15807 raft_consensus.cc:2726] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 06:57:39.455178 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:39.454399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:52249)
0927 06:57:39.454825 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:39.454936 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":211,"threads_started":1}
I20260927 06:57:39.603822 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:39.603411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:33393)
0927 06:57:39.603572 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:39.603676 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 06:57:40.033931 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:40.033189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:56819)
0927 06:57:40.033654 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:40.033742 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":246,"threads_started":1}
I20260927 06:57:40.456624 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:40.455995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:51381)
0927 06:57:40.456406 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:40.456477 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":215,"threads_started":1}
I20260927 06:57:40.605337 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:40.604630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:45353)
0927 06:57:40.605086 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:40.605163 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":213,"threads_started":1}
I20260927 06:57:41.035497 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:41.034758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:34391)
0927 06:57:41.035162 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:41.035262 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":236,"threads_started":1}
I20260927 06:57:41.120906 15817 raft_consensus.cc:482] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:57:41.121078 15817 raft_consensus.cc:504] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } } peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } }
I20260927 06:57:41.121678 15817 leader_election.cc:288] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 454b008c6926463e9778ee64e8c06a84 (127.0.97.3:44137), 8931a6ee797b43d291c219d5f0b1404e (127.0.97.2:44123)
W20260927 06:57:41.122599 15425 leader_election.cc:341] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 454b008c6926463e9778ee64e8c06a84 (127.0.97.3:44137): Not found: Tablet not found: 9c1cd5e2e1c241de9115e0e82232d62b
W20260927 06:57:41.122854 15425 leader_election.cc:341] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8931a6ee797b43d291c219d5f0b1404e (127.0.97.2:44123): Not found: Tablet not found: 9c1cd5e2e1c241de9115e0e82232d62b
I20260927 06:57:41.122962 15425 leader_election.cc:302] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01c30625ed884c11959092b7148a8239; no voters: 454b008c6926463e9778ee64e8c06a84, 8931a6ee797b43d291c219d5f0b1404e
I20260927 06:57:41.123199 15817 raft_consensus.cc:2726] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 06:57:41.458130 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:41.457468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:58407)
0927 06:57:41.457874 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:41.457941 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":215,"threads_started":1}
I20260927 06:57:41.606951 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:41.606275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:49779)
0927 06:57:41.606675 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:41.606771 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":197,"threads_started":1}
I20260927 06:57:42.037042 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:42.036368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:35935)
0927 06:57:42.036747 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:42.036836 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":189,"threads_started":1}
I20260927 06:57:42.459496 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:42.458885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:41977)
0927 06:57:42.459254 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:42.459328 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":177,"threads_started":1}
I20260927 06:57:42.608606 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:42.607955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:36669)
0927 06:57:42.608360 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:42.608443 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":209,"threads_started":1}
I20260927 06:57:43.038589 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:43.037936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:50163)
0927 06:57:43.038365 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:43.038436 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":250,"threads_started":1}
I20260927 06:57:43.443063 15824 raft_consensus.cc:482] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:57:43.443260 15824 raft_consensus.cc:504] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } } peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } }
I20260927 06:57:43.443868 15824 leader_election.cc:288] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 454b008c6926463e9778ee64e8c06a84 (127.0.97.3:44137), 8931a6ee797b43d291c219d5f0b1404e (127.0.97.2:44123)
W20260927 06:57:43.444802 15425 leader_election.cc:341] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 454b008c6926463e9778ee64e8c06a84 (127.0.97.3:44137): Not found: Tablet not found: 9c1cd5e2e1c241de9115e0e82232d62b
W20260927 06:57:43.444983 15425 leader_election.cc:341] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8931a6ee797b43d291c219d5f0b1404e (127.0.97.2:44123): Not found: Tablet not found: 9c1cd5e2e1c241de9115e0e82232d62b
I20260927 06:57:43.445070 15425 leader_election.cc:302] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01c30625ed884c11959092b7148a8239; no voters: 454b008c6926463e9778ee64e8c06a84, 8931a6ee797b43d291c219d5f0b1404e
I20260927 06:57:43.445282 15824 raft_consensus.cc:2726] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 06:57:43.461066 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:43.460362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:51377)
0927 06:57:43.460846 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:43.460917 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":284,"threads_started":1}
I20260927 06:57:43.610268 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:43.609498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:48891)
0927 06:57:43.609934 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:43.610005 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":252,"threads_started":1}
I20260927 06:57:44.040072 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:44.039407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:46115)
0927 06:57:44.039828 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:44.039901 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":219,"threads_started":1}
I20260927 06:57:44.462627 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:44.461799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:48841)
0927 06:57:44.462300 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:44.462439 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":314,"threads_started":1}
I20260927 06:57:44.611997 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:44.611273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:53559)
0927 06:57:44.611690 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:44.611765 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":231,"threads_started":1}
I20260927 06:57:45.041765 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:45.041013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:49631)
0927 06:57:45.041509 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:45.041585 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":223,"threads_started":1}
I20260927 06:57:45.464080 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:45.463383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:47039)
0927 06:57:45.463820 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:45.463890 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":209,"threads_started":1}
I20260927 06:57:45.613631 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:45.612887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:55531)
0927 06:57:45.613388 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:45.613487 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":274,"threads_started":1}
I20260927 06:57:46.043347 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:46.042694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:41095)
0927 06:57:46.043111 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:46.043186 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":221,"threads_started":1}
I20260927 06:57:46.465734 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:46.465053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:43169)
0927 06:57:46.465466 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:46.465559 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":222,"threads_started":1}
I20260927 06:57:46.615097 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:46.614458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:43625)
0927 06:57:46.614863 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:46.614937 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":221,"threads_started":1}
I20260927 06:57:47.044888 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:47.044194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:56513)
0927 06:57:47.044635 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:47.044727 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":207,"threads_started":1}
I20260927 06:57:47.467445 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:47.466739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:57449)
0927 06:57:47.467214 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:47.467290 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":226,"threads_started":1}
I20260927 06:57:47.616621 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:47.615978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:44485)
0927 06:57:47.616387 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:47.616466 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":223,"threads_started":1}
I20260927 06:57:47.962591 15839 raft_consensus.cc:482] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:57:47.962898 15839 raft_consensus.cc:504] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454b008c6926463e9778ee64e8c06a84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44137 } } peers { permanent_uuid: "01c30625ed884c11959092b7148a8239" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40123 } } peers { permanent_uuid: "8931a6ee797b43d291c219d5f0b1404e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44123 } }
I20260927 06:57:47.963691 15839 leader_election.cc:288] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 454b008c6926463e9778ee64e8c06a84 (127.0.97.3:44137), 8931a6ee797b43d291c219d5f0b1404e (127.0.97.2:44123)
W20260927 06:57:47.964602 15425 leader_election.cc:341] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 454b008c6926463e9778ee64e8c06a84 (127.0.97.3:44137): Not found: Tablet not found: 9c1cd5e2e1c241de9115e0e82232d62b
W20260927 06:57:47.964815 15425 leader_election.cc:341] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8931a6ee797b43d291c219d5f0b1404e (127.0.97.2:44123): Not found: Tablet not found: 9c1cd5e2e1c241de9115e0e82232d62b
I20260927 06:57:47.964934 15425 leader_election.cc:302] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01c30625ed884c11959092b7148a8239; no voters: 454b008c6926463e9778ee64e8c06a84, 8931a6ee797b43d291c219d5f0b1404e
I20260927 06:57:47.965195 15839 raft_consensus.cc:2726] T 9c1cd5e2e1c241de9115e0e82232d62b P 01c30625ed884c11959092b7148a8239 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 06:57:48.046478 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:48.045773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:54805)
0927 06:57:48.046238 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:48.046310 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":263,"threads_started":1}
I20260927 06:57:48.468873 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:48.468285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:53809)
0927 06:57:48.468651 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:48.468715 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":195,"threads_started":1}
I20260927 06:57:48.618285 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:48.617544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:52305)
0927 06:57:48.617941 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:48.618070 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":203,"threads_started":1}
I20260927 06:57:49.048102 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:49.047428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:58393)
0927 06:57:49.047862 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:49.047933 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":207,"threads_started":1}
I20260927 06:57:49.470513 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:49.469758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:56167)
0927 06:57:49.470273 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:49.470351 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":293,"threads_started":1}
I20260927 06:57:49.619763 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:49.619103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:54441)
0927 06:57:49.619512 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:49.619590 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":224,"threads_started":1}
I20260927 06:57:50.049597 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:50.048917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:46715)
0927 06:57:50.049354 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:50.049427 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":229,"threads_started":1}
I20260927 06:57:50.471987 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:50.471298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:38457)
0927 06:57:50.471741 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:50.471813 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":245,"threads_started":1}
I20260927 06:57:50.621155 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:50.620526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:41375)
0927 06:57:50.620930 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:50.621000 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":219,"threads_started":1}
I20260927 06:57:51.051218 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:51.050578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:40955)
0927 06:57:51.050988 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:51.051062 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":210,"threads_started":1}
I20260927 06:57:51.473591 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:51.472915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:53591)
0927 06:57:51.473362 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:51.473433 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":224,"threads_started":1}
I20260927 06:57:51.622691 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:51.622036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:57913)
0927 06:57:51.622435 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:51.622509 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":201,"threads_started":1}
I20260927 06:57:52.052752 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:52.052051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:57513)
0927 06:57:52.052483 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:52.052559 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":218,"threads_started":1}
I20260927 06:57:52.475163 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:52.474469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:55245)
0927 06:57:52.474894 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:52.474967 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":214,"threads_started":1}
I20260927 06:57:52.624388 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:52.623719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.1:37515)
0927 06:57:52.624146 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:52.624218 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":221,"threads_started":1}
I20260927 06:57:53.054742 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:53.053960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.2:39265)
0927 06:57:53.054507 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:53.054599 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":331,"threads_started":1}
I20260927 06:57:53.458863 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 15373
I20260927 06:57:53.476889 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:53.476159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36617 (local address 127.0.97.3:36141)
0927 06:57:53.476643 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:53.476733 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":251,"threads_started":1}
I20260927 06:57:53.478528 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:53.459897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54976 (local address 127.0.97.1:40123)
0927 06:57:53.460550 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:53.460564 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:53.460599 (+ 35us) server_negotiation.cc:407] Connection header received
0927 06:57:53.460686 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:53.460695 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:53.460941 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:53.461087 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:53.462477 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:53.463440 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:53.464696 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:53.465403 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:53.465501 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:53.472904 (+ 7403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:53.472917 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:53.472931 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:53.472967 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:53.475671 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:53.476390 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:53.476405 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:53.476408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:53.476546 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:53.476926 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:53.476932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:53.476935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:53.477318 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:53.477624 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:53.477938 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 06:57:53.478230 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":494,"threads_started":1}
I20260927 06:57:53.479736 15533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:53.566968 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 15373
I20260927 06:57:53.584183 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 15542
I20260927 06:57:53.597184 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:53.584898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45268 (local address 127.0.97.2:44123)
0927 06:57:53.585409 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:53.585415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:53.585430 (+ 15us) server_negotiation.cc:407] Connection header received
0927 06:57:53.585483 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:53.585486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:53.585543 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:53.585660 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:53.587052 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:53.587834 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:53.589096 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:53.589545 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:53.589683 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:53.592197 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:53.592202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:53.592205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:53.592236 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:53.594603 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:53.595258 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:53.595265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:53.595268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:53.595400 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:53.595791 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:53.595800 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:53.595802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:53.596050 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:53.596325 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:53.596593 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 06:57:53.596814 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":250,"threads_started":1}
I20260927 06:57:53.597852 15665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:53.679260 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 15542
I20260927 06:57:53.696897 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 15673
I20260927 06:57:53.708997 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:53.697677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.0.97.3:44137)
0927 06:57:53.698211 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:53.698216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:53.698232 (+ 16us) server_negotiation.cc:407] Connection header received
0927 06:57:53.698286 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:53.698290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:53.698354 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:53.698442 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:53.699658 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:53.700433 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:53.701550 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:53.701936 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:53.702170 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:53.704322 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:53.704327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:53.704330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:53.704358 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:53.706818 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:53.707331 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:53.707334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:53.707336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:53.707483 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:53.707782 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:53.707785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:53.707787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:53.708003 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:53.708191 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:53.708399 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 06:57:53.708554 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":251,"threads_started":1}
I20260927 06:57:53.709637 15796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:53.790589 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 15673
2026-09-27T06:57:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23996 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:57:53 dist-test-slave-f892 krb5kdc[15865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:57:53 dist-test-slave-f892 krb5kdc[15865](info): set up 2 sockets
Sep 27 06:57:53 dist-test-slave-f892 krb5kdc[15865](info): commencing operation
krb5kdc: starting...
W20260927 06:57:54.880118 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:57:55 dist-test-slave-f892 krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:57:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:57:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:57:55.099606 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45499
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:57:55.414927 15878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:55.415194 15878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:55.415385 15878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:55.422611 15878 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 06:57:55.422710 15878 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 06:57:55.422797 15878 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 06:57:55.422904 15878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:57:55.423013 15878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:55.423113 15878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:55.423200 15878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:57:55.423278 15878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:57:55.433045 15878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:55.434321 15878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:55.435631 15878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:55.447436 15886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:55.447438 15883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:55.447463 15884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:55.447757 15878 server_base.cc:1034] running on GCE node
I20260927 06:57:55.448825 15878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:55.449997 15878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:55.451270 15878 hybrid_clock.cc:648] HybridClock initialized: now 1790492275451213 us; error 27 us; skew 500 ppm
I20260927 06:57:55.451609 15878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:55 dist-test-slave-f892 krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:55.454700 15878 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:57:55.456012 15878 webserver.cc:460] Webserver started at http://127.0.97.62:45477/ using document root <none> and password file <none>
I20260927 06:57:55.456653 15878 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:55.456782 15878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:55.457096 15878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:57:55.459880 15878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "ef828650a9aa458aa7e0029f55175129"
format_stamp: "Formatted at 2026-09-27 06:57:55 on dist-test-slave-f892"
server_key: "83061500fde877d5db55a5c936037449"
server_key_iv: "9f3075a826dfe19255dde3560a6c1894"
server_key_version: "encryptionkey@0"
I20260927 06:57:55.460623 15878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "ef828650a9aa458aa7e0029f55175129"
format_stamp: "Formatted at 2026-09-27 06:57:55 on dist-test-slave-f892"
server_key: "83061500fde877d5db55a5c936037449"
server_key_iv: "9f3075a826dfe19255dde3560a6c1894"
server_key_version: "encryptionkey@0"
I20260927 06:57:55.465205 15878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 06:57:55.468384 15893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:55.469595 15878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 06:57:55.469769 15878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "ef828650a9aa458aa7e0029f55175129"
format_stamp: "Formatted at 2026-09-27 06:57:55 on dist-test-slave-f892"
server_key: "83061500fde877d5db55a5c936037449"
server_key_iv: "9f3075a826dfe19255dde3560a6c1894"
server_key_version: "encryptionkey@0"
I20260927 06:57:55.469987 15878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:55.483052 15878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:55.487121 15878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:55.487411 15878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:55.504357 15878 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45499
I20260927 06:57:55.504376 15944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45499 every 8 connection(s)
I20260927 06:57:55.505556 15878 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:57:55.509128 15945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:55.509644 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 15878
I20260927 06:57:55.509863 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:57:55.510571 388 external_mini_cluster.cc:1428] Setting key a92c3f2ad7c25dfff17f8fe31c295e63
I20260927 06:57:55.520617 15945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: Bootstrap starting.
Sep 27 06:57:55 dist-test-slave-f892 krb5kdc[15865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:55.523566 15945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: Neither blocks nor log segments found. Creating new log.
I20260927 06:57:55.524505 15945 log.cc:824] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:55.527351 15945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: No bootstrap required, opened a new log
I20260927 06:57:55.531504 15948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:55.512477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41774 (local address 127.0.97.62:45499)
0927 06:57:55.513229 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:55.513239 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:55.513278 (+ 39us) server_negotiation.cc:407] Connection header received
0927 06:57:55.514858 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:55.514878 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:55.515221 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:55.515605 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:55.517007 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:55.518341 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:55.519642 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:55.520271 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:55.520458 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:55.524357 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:55.524364 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:55.524376 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:55.524408 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:55.527658 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:55.528279 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:55.528292 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:55.528303 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:55.528456 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:55.528823 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:55.528834 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:55.528837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:55.529410 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:55.529749 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:55.530183 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 06:57:55.530559 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":408,"threads_started":1}
I20260927 06:57:55.535372 15945 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef828650a9aa458aa7e0029f55175129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260927 06:57:55.535748 15945 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:55.535851 15945 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef828650a9aa458aa7e0029f55175129, State: Initialized, Role: FOLLOWER
I20260927 06:57:55.536413 15945 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef828650a9aa458aa7e0029f55175129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260927 06:57:55.536648 15945 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:57:55.536795 15945 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:57:55.536967 15945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:55.539085 15945 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef828650a9aa458aa7e0029f55175129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260927 06:57:55.539541 15945 leader_election.cc:302] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef828650a9aa458aa7e0029f55175129; no voters:
I20260927 06:57:55.540064 15945 leader_election.cc:288] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:57:55.540182 15950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:57:55.540863 15950 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 1 LEADER]: Becoming Leader. State: Replica: ef828650a9aa458aa7e0029f55175129, State: Running, Role: LEADER
I20260927 06:57:55.541321 15950 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef828650a9aa458aa7e0029f55175129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260927 06:57:55.541954 15945 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:57:55.544070 15952 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef828650a9aa458aa7e0029f55175129. Latest consensus state: current_term: 1 leader_uuid: "ef828650a9aa458aa7e0029f55175129" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef828650a9aa458aa7e0029f55175129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } }
I20260927 06:57:55.544478 15952 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [sys.catalog]: This master's current role is: LEADER
I20260927 06:57:55.544045 15951 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef828650a9aa458aa7e0029f55175129" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef828650a9aa458aa7e0029f55175129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } }
I20260927 06:57:55.545114 15951 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [sys.catalog]: This master's current role is: LEADER
I20260927 06:57:55.545336 15956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:57:55.550510 15956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:57:55.560634 15956 catalog_manager.cc:1347] Generated new cluster ID: 4c2bb57af5bf40d98628297bd2f6a74f
I20260927 06:57:55.560761 15956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:57:55.573019 15956 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:57:55.574328 15956 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:57:55.580236 15956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: Generated new TSK 0
I20260927 06:57:55.581059 15956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 06:57:55.637316 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45499
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:57:55.954205 15971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:55.954483 15971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:55.954645 15971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:55.961575 15971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:55.961719 15971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:55.961897 15971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:57:55.971426 15971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:55.972715 15971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:55.974134 15971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:55.987353 15979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:55.987432 15976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:55.987409 15977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:55.988546 15971 server_base.cc:1034] running on GCE node
I20260927 06:57:55.989106 15971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:55.989957 15971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:55.991206 15971 hybrid_clock.cc:648] HybridClock initialized: now 1790492275991163 us; error 35 us; skew 500 ppm
I20260927 06:57:55.991528 15971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:55 dist-test-slave-f892 krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:55.994619 15971 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:57:55.996028 15971 webserver.cc:460] Webserver started at http://127.0.97.1:37755/ using document root <none> and password file <none>
I20260927 06:57:55.996659 15971 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:55.996799 15971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:55.997107 15971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:57:55.999876 15971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "bc2c9c376ab7450fbd7893fe5112ec48"
format_stamp: "Formatted at 2026-09-27 06:57:55 on dist-test-slave-f892"
server_key: "25459d9c65c7f2354730f04a0b216abf"
server_key_iv: "0af4105c19f416eccdde695320cf6d8e"
server_key_version: "encryptionkey@0"
I20260927 06:57:56.000609 15971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bc2c9c376ab7450fbd7893fe5112ec48"
format_stamp: "Formatted at 2026-09-27 06:57:55 on dist-test-slave-f892"
server_key: "25459d9c65c7f2354730f04a0b216abf"
server_key_iv: "0af4105c19f416eccdde695320cf6d8e"
server_key_version: "encryptionkey@0"
I20260927 06:57:56.005316 15971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 06:57:56.008612 15986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:56.009950 15971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 06:57:56.010134 15971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "bc2c9c376ab7450fbd7893fe5112ec48"
format_stamp: "Formatted at 2026-09-27 06:57:55 on dist-test-slave-f892"
server_key: "25459d9c65c7f2354730f04a0b216abf"
server_key_iv: "0af4105c19f416eccdde695320cf6d8e"
server_key_version: "encryptionkey@0"
I20260927 06:57:56.010311 15971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:56.049170 15971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:56.053045 15971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:56.053284 15971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:56.054219 15971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:57:56.055509 15971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:57:56.055588 15971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:56.055670 15971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:57:56.055716 15971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:56.087589 15971 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45395
I20260927 06:57:56.087641 16098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45395 every 8 connection(s)
I20260927 06:57:56.088939 15971 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:57:56.090598 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 15971
I20260927 06:57:56.090819 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:57:56.091517 388 external_mini_cluster.cc:1428] Setting key 0f6fb7b64fedd81f6d1ada60210b4095
Sep 27 06:57:56 dist-test-slave-f892 krb5kdc[15865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:56.112277 16101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:56.091594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42449 (local address 127.0.97.62:45499)
0927 06:57:56.092070 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:56.092076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:56.093188 (+ 1112us) server_negotiation.cc:407] Connection header received
0927 06:57:56.095840 (+ 2652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:56.095844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:56.095908 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:56.095997 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:56.097992 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:56.098753 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:56.101191 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:56.101623 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:56.101719 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:56.105449 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:56.105459 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:56.105467 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:56.105511 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:56.109053 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:56.109724 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:56.109733 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:56.109739 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:56.109890 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:56.110335 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:56.110342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:56.110348 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:56.110719 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:56.110870 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:56.111542 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 06:57:56.111792 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":224,"threads_started":1}
I20260927 06:57:56.112848 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:56.092187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:42449)
0927 06:57:56.093011 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:56.093068 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:56.095532 (+ 2464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:56.096255 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:56.096268 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:56.097048 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:56.097827 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:56.097852 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:56.098915 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:56.098921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:56.101044 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:56.101061 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:56.101853 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:56.101863 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:56.102198 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:56.102907 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:56.102932 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:56.105230 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:56.109199 (+ 3969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:56.109205 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:56.109217 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:56.109607 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:56.109997 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:56.110000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:56.110002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:56.110226 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:56.110843 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:56.110849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:56.111136 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 06:57:56.111365 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 06:57:56.111731 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":339,"threads_started":1}
I20260927 06:57:56.114590 16099 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45499
I20260927 06:57:56.114955 16099 heartbeater.cc:464] Registering TS with master...
I20260927 06:57:56.115864 16099 heartbeater.cc:511] Master 127.0.97.62:45499 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 06:57:56.118358 15910 ts_manager.cc:194] Registered new tserver with Master: bc2c9c376ab7450fbd7893fe5112ec48 (127.0.97.1:45395)
I20260927 06:57:56.119838 15910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 06:57:56.148801 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45499
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:57:56.464174 16105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:56.464433 16105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:56.464612 16105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:56.471647 16105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:56.471783 16105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:56.471967 16105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:57:56.481472 16105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:56.482769 16105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:56.484063 16105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:56.497107 16111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:56.497210 16110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:56.497445 16113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:56.497839 16105 server_base.cc:1034] running on GCE node
I20260927 06:57:56.498521 16105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:56.499325 16105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:56.500512 16105 hybrid_clock.cc:648] HybridClock initialized: now 1790492276500481 us; error 23 us; skew 500 ppm
I20260927 06:57:56.500756 16105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:56 dist-test-slave-f892 krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:56.503676 16105 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:57:56.505127 16105 webserver.cc:460] Webserver started at http://127.0.97.2:40523/ using document root <none> and password file <none>
I20260927 06:57:56.505748 16105 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:56.505888 16105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:56.506223 16105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:57:56.508985 16105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "313768f6bc634745b93a16c93271bb36"
format_stamp: "Formatted at 2026-09-27 06:57:56 on dist-test-slave-f892"
server_key: "cfa03080971567a9aa0ad46b37b8dca8"
server_key_iv: "4c6913969f997a7addb7e699620f6a91"
server_key_version: "encryptionkey@0"
I20260927 06:57:56.509701 16105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "313768f6bc634745b93a16c93271bb36"
format_stamp: "Formatted at 2026-09-27 06:57:56 on dist-test-slave-f892"
server_key: "cfa03080971567a9aa0ad46b37b8dca8"
server_key_iv: "4c6913969f997a7addb7e699620f6a91"
server_key_version: "encryptionkey@0"
I20260927 06:57:56.515105 16105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 06:57:56.518555 16120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:56.519857 16105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 06:57:56.520015 16105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "313768f6bc634745b93a16c93271bb36"
format_stamp: "Formatted at 2026-09-27 06:57:56 on dist-test-slave-f892"
server_key: "cfa03080971567a9aa0ad46b37b8dca8"
server_key_iv: "4c6913969f997a7addb7e699620f6a91"
server_key_version: "encryptionkey@0"
I20260927 06:57:56.520203 16105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:56.536202 16105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:56.539757 16105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:56.540032 16105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:56.540897 16105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:57:56.542300 16105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:57:56.542397 16105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:56.542552 16105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:57:56.542620 16105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:56.575129 16105 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38027
I20260927 06:57:56.575170 16232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38027 every 8 connection(s)
I20260927 06:57:56.576373 16105 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:57:56.579509 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 16105
I20260927 06:57:56.579727 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:57:56.580335 388 external_mini_cluster.cc:1428] Setting key e58a1aaabd3f4d838020fe411d92f682
Sep 27 06:57:56 dist-test-slave-f892 krb5kdc[15865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:56.597378 16101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:56.578932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45437 (local address 127.0.97.62:45499)
0927 06:57:56.579117 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:56.579122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:56.580536 (+ 1414us) server_negotiation.cc:407] Connection header received
0927 06:57:56.582484 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:56.582491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:56.582576 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:56.582680 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:56.584204 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:56.585340 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:56.586920 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:56.587663 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:56.587822 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:56.591829 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:56.591834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:56.591837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:56.591867 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:56.594348 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:56.595229 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:56.595232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:56.595234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:56.595358 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:56.595827 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:56.595830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:56.595832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:56.596077 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:56.596186 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:56.596987 (+ 801us) server_negotiation.cc:299] Negotiation successful
0927 06:57:56.597153 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 06:57:56.598913 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:56.579524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:45437)
0927 06:57:56.580361 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:56.580414 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:56.582205 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:56.582983 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:56.583000 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:56.583491 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:56.584023 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:56.584040 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:56.585490 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:56.585493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:56.586778 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:56.586790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:56.587943 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:56.587952 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:56.588202 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:56.588882 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:56.588904 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:56.591620 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:56.594475 (+ 2855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:56.594486 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:56.594504 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:56.595115 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:56.595462 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:56.595469 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:56.595475 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:56.595711 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:56.596203 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:56.596214 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:56.596621 (+ 407us) client_negotiation.cc:769] Sending connection context
0927 06:57:56.596925 (+ 304us) client_negotiation.cc:240] Negotiation successful
0927 06:57:56.597451 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":329,"threads_started":1}
I20260927 06:57:56.600299 16233 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45499
I20260927 06:57:56.600584 16233 heartbeater.cc:464] Registering TS with master...
I20260927 06:57:56.601250 16233 heartbeater.cc:511] Master 127.0.97.62:45499 requested a full tablet report, sending...
I20260927 06:57:56.602713 15910 ts_manager.cc:194] Registered new tserver with Master: 313768f6bc634745b93a16c93271bb36 (127.0.97.2:38027)
I20260927 06:57:56.603899 15910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45437
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 06:57:56.632589 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45499
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:57:56.951849 16238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:57:56.952099 16238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:57:56.952189 16238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:57:56.959154 16238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:57:56.959251 16238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:57:56.959357 16238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:57:56.968976 16238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:57:56.970287 16238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:57:56.971580 16238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:57:56.985446 16244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:56.985517 16243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:57:56.985486 16246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:57:56.986362 16238 server_base.cc:1034] running on GCE node
I20260927 06:57:56.986929 16238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:57:56.987823 16238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:57:56.989046 16238 hybrid_clock.cc:648] HybridClock initialized: now 1790492276989002 us; error 37 us; skew 500 ppm
I20260927 06:57:56.989341 16238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:57:56 dist-test-slave-f892 krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:57:56.992451 16238 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:57:56.993724 16238 webserver.cc:460] Webserver started at http://127.0.97.3:38371/ using document root <none> and password file <none>
I20260927 06:57:56.994525 16238 fs_manager.cc:359] Metadata directory not provided
I20260927 06:57:56.994626 16238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:57:56.994870 16238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:57:56.997911 16238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "66daad4cb9374f83a86b311ad2cf6272"
format_stamp: "Formatted at 2026-09-27 06:57:56 on dist-test-slave-f892"
server_key: "93f3078ed97acb1930c71cfbf83b00ab"
server_key_iv: "4f590b0ec4fe35629c42e4b069a71025"
server_key_version: "encryptionkey@0"
I20260927 06:57:56.998690 16238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "66daad4cb9374f83a86b311ad2cf6272"
format_stamp: "Formatted at 2026-09-27 06:57:56 on dist-test-slave-f892"
server_key: "93f3078ed97acb1930c71cfbf83b00ab"
server_key_iv: "4f590b0ec4fe35629c42e4b069a71025"
server_key_version: "encryptionkey@0"
I20260927 06:57:57.003547 16238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 06:57:57.006879 16253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:57.008118 16238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 06:57:57.008239 16238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "66daad4cb9374f83a86b311ad2cf6272"
format_stamp: "Formatted at 2026-09-27 06:57:56 on dist-test-slave-f892"
server_key: "93f3078ed97acb1930c71cfbf83b00ab"
server_key_iv: "4f590b0ec4fe35629c42e4b069a71025"
server_key_version: "encryptionkey@0"
I20260927 06:57:57.008388 16238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:57:57.023923 16238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:57:57.027936 16238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:57:57.028168 16238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:57:57.029042 16238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:57:57.030558 16238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:57:57.030645 16238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:57.030740 16238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:57:57.030787 16238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:57:57.063860 16238 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42465
I20260927 06:57:57.063946 16365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42465 every 8 connection(s)
I20260927 06:57:57.065114 16238 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:57:57.073750 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 16238
I20260927 06:57:57.073976 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:57:57.074563 388 external_mini_cluster.cc:1428] Setting key b9d92da4f350e1331aed36d1d2112a81
Sep 27 06:57:57 dist-test-slave-f892 krb5kdc[15865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:57:57.085063 16101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:57.067652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56231 (local address 127.0.97.62:45499)
0927 06:57:57.067894 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:57.067900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:57.068911 (+ 1011us) server_negotiation.cc:407] Connection header received
0927 06:57:57.070693 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:57.070697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:57.070780 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:57.070866 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:57.072245 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.073117 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.075425 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.076017 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.076124 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:57.079826 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:57.079830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:57.079833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:57.079866 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:57.082618 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:57.083246 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:57.083249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:57.083251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:57.083365 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:57.083711 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:57.083714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:57.083716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:57.083953 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:57.084059 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:57.084654 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 06:57:57.084806 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260927 06:57:57.085934 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:57.068089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:56231)
0927 06:57:57.068756 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:57.068790 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:57.070468 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:57.071062 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:57.071070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:57.071577 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:57:57.072109 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:57.072125 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.073246 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.073250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:57.075251 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:57.075262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.076249 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.076257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:57.076549 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:57.077317 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:57:57.077351 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:57:57.079634 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:57:57.082735 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:57.082741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:57.082752 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:57.083144 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:57.083451 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:57:57.083454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:57:57.083456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:57:57.083619 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:57:57.084058 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:57:57.084076 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:57:57.084344 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 06:57:57.084548 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 06:57:57.084884 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":273,"threads_started":1}
I20260927 06:57:57.087623 16366 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45499
I20260927 06:57:57.087908 16366 heartbeater.cc:464] Registering TS with master...
I20260927 06:57:57.088755 16366 heartbeater.cc:511] Master 127.0.97.62:45499 requested a full tablet report, sending...
I20260927 06:57:57.090346 15910 ts_manager.cc:194] Registered new tserver with Master: 66daad4cb9374f83a86b311ad2cf6272 (127.0.97.3:42465)
I20260927 06:57:57.091228 15910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56231
I20260927 06:57:57.101867 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:57:57.118266 16101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:57.106305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41780 (local address 127.0.97.62:45499)
0927 06:57:57.106460 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:57.106464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:57.106886 (+ 422us) server_negotiation.cc:407] Connection header received
0927 06:57:57.107009 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:57.107012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:57.107074 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:57.107151 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:57.108503 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.109196 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.110574 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.110948 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.111039 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:57.113465 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:57.113469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:57.113472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:57.113502 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:57.115934 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:57.116572 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:57.116576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:57.116578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:57.116686 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:57.116977 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:57.116980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:57.116982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:57.117266 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:57.117439 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:57.117729 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 06:57:57.117889 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 06:57:57.123334 16099 heartbeater.cc:503] Master 127.0.97.62:45499 was elected leader, sending a full tablet report...
I20260927 06:57:57.121726 15910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:57:57.129082 15910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:57:57.154177 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:57.140445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.0.97.3:42465)
0927 06:57:57.141185 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:57.141195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:57.141907 (+ 712us) server_negotiation.cc:407] Connection header received
0927 06:57:57.142317 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:57.142326 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:57.142561 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:57.142721 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:57.143894 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.146047 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.149620 (+ 3573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.152325 (+ 2705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.152456 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:57.152739 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:57.153600 (+ 861us) server_negotiation.cc:299] Negotiation successful
0927 06:57:57.153782 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":424,"threads_started":1}
I20260927 06:57:57.155055 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:57.140146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42465 (local address 127.0.0.1:52540)
0927 06:57:57.141761 (+ 1615us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:57.141798 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:57.142040 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:57.143082 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:57.143088 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:57.143115 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:57.143625 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:57.143651 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.146278 (+ 2627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.146286 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:57.149401 (+ 3115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:57.149421 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.152956 (+ 3535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.152964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:57.153263 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:57.153343 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 06:57:57.154437 (+ 1094us) client_negotiation.cc:240] Negotiation successful
0927 06:57:57.154668 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1371,"mutex_wait_us":1,"thread_start_us":2474,"threads_started":1}
I20260927 06:57:57.156805 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:57.140393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45395 (local address 127.0.0.1:48752)
0927 06:57:57.141821 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:57.141843 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:57.142040 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:57.143289 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:57.143302 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:57.143338 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:57.144078 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:57.144094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.147984 (+ 3890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.147992 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:57.152252 (+ 4260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:57.152273 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.156034 (+ 3761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.156040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:57.156237 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:57.156267 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 06:57:57.156367 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 06:57:57.156499 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1281,"thread_start_us":1485,"threads_started":1}
I20260927 06:57:57.156934 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:57.140850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48752 (local address 127.0.97.1:45395)
0927 06:57:57.141494 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:57.141503 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:57.142247 (+ 744us) server_negotiation.cc:407] Connection header received
0927 06:57:57.142363 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:57.142374 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:57.142709 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:57.142933 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:57.144336 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.147790 (+ 3454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.152509 (+ 4719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.155815 (+ 3306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.155960 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:57.156175 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:57.156427 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 06:57:57.156584 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":1302,"threads_started":1}
I20260927 06:57:57.158866 16301 tablet_service.cc:1467] Processing CreateTablet for tablet 68a7bb949a464d59ba15e41231a5c888 (DEFAULT_TABLE table=test-table [id=47dbc8802be8445cb4a9c36e1afd1c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:57:57.160378 16301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a7bb949a464d59ba15e41231a5c888. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:57.160223 16034 tablet_service.cc:1467] Processing CreateTablet for tablet 68a7bb949a464d59ba15e41231a5c888 (DEFAULT_TABLE table=test-table [id=47dbc8802be8445cb4a9c36e1afd1c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:57:57.161499 16034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a7bb949a464d59ba15e41231a5c888. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:57.162966 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:57.141716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38027 (local address 127.0.0.1:38820)
0927 06:57:57.153848 (+ 12132us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:57.153867 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:57.153987 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:57.155669 (+ 1682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:57.155674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:57.155699 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:57.156068 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:57.156083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.157972 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.157977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:57.160348 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:57.160364 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.162226 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.162232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:57.162386 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:57.162417 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 06:57:57.162521 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 06:57:57.162643 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12013,"thread_start_us":12366,"threads_started":1}
I20260927 06:57:57.163060 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:57.142023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.0.97.2:38027)
0927 06:57:57.155171 (+ 13148us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:57.155182 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:57.155218 (+ 36us) server_negotiation.cc:407] Connection header received
0927 06:57:57.155312 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:57.155322 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:57.155505 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:57.155644 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:57.156273 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.157824 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.160559 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.162041 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.162166 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:57.162326 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:57.162581 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 06:57:57.162732 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12974,"thread_start_us":12842,"threads_started":1}
I20260927 06:57:57.166500 16168 tablet_service.cc:1467] Processing CreateTablet for tablet 68a7bb949a464d59ba15e41231a5c888 (DEFAULT_TABLE table=test-table [id=47dbc8802be8445cb4a9c36e1afd1c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:57:57.167774 16168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a7bb949a464d59ba15e41231a5c888. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:57:57.173112 16384 tablet_bootstrap.cc:492] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48: Bootstrap starting.
I20260927 06:57:57.175071 16385 tablet_bootstrap.cc:492] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272: Bootstrap starting.
I20260927 06:57:57.176826 16384 tablet_bootstrap.cc:654] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48: Neither blocks nor log segments found. Creating new log.
I20260927 06:57:57.178081 16384 log.cc:824] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:57.178509 16385 tablet_bootstrap.cc:654] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272: Neither blocks nor log segments found. Creating new log.
I20260927 06:57:57.179615 16385 log.cc:824] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:57.181504 16384 tablet_bootstrap.cc:492] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48: No bootstrap required, opened a new log
I20260927 06:57:57.181823 16384 ts_tablet_manager.cc:1397] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 06:57:57.183506 16385 tablet_bootstrap.cc:492] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272: No bootstrap required, opened a new log
I20260927 06:57:57.183656 16388 tablet_bootstrap.cc:492] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36: Bootstrap starting.
I20260927 06:57:57.183749 16385 ts_tablet_manager.cc:1397] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260927 06:57:57.186249 16388 tablet_bootstrap.cc:654] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36: Neither blocks nor log segments found. Creating new log.
I20260927 06:57:57.187109 16388 log.cc:824] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36: Log is configured to *not* fsync() on all Append() calls
I20260927 06:57:57.191560 16388 tablet_bootstrap.cc:492] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36: No bootstrap required, opened a new log
I20260927 06:57:57.191794 16388 ts_tablet_manager.cc:1397] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 06:57:57.192277 16384 raft_consensus.cc:348] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } } peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } }
I20260927 06:57:57.192672 16384 raft_consensus.cc:374] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:57.192770 16384 raft_consensus.cc:729] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc2c9c376ab7450fbd7893fe5112ec48, State: Initialized, Role: FOLLOWER
I20260927 06:57:57.193373 16384 consensus_queue.cc:260] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } } peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } }
I20260927 06:57:57.194315 16385 raft_consensus.cc:348] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } } peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } }
I20260927 06:57:57.194819 16385 raft_consensus.cc:374] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:57.194936 16385 raft_consensus.cc:729] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66daad4cb9374f83a86b311ad2cf6272, State: Initialized, Role: FOLLOWER
I20260927 06:57:57.195603 16385 consensus_queue.cc:260] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } } peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } }
I20260927 06:57:57.199548 16384 ts_tablet_manager.cc:1428] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48: Time spent starting tablet: real 0.018s user 0.009s sys 0.007s
I20260927 06:57:57.199352 16388 raft_consensus.cc:348] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } } peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } }
I20260927 06:57:57.199712 16388 raft_consensus.cc:374] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:57:57.199855 16388 raft_consensus.cc:729] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313768f6bc634745b93a16c93271bb36, State: Initialized, Role: FOLLOWER
I20260927 06:57:57.200580 16366 heartbeater.cc:503] Master 127.0.97.62:45499 was elected leader, sending a full tablet report...
I20260927 06:57:57.200451 16388 consensus_queue.cc:260] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } } peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } }
I20260927 06:57:57.201462 16233 heartbeater.cc:503] Master 127.0.97.62:45499 was elected leader, sending a full tablet report...
I20260927 06:57:57.202128 16388 ts_tablet_manager.cc:1428] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260927 06:57:57.203512 16385 ts_tablet_manager.cc:1428] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272: Time spent starting tablet: real 0.020s user 0.010s sys 0.007s
W20260927 06:57:57.316783 16367 tablet.cc:2369] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:57:57.328241 16234 tablet.cc:2369] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:57:57.341317 16100 tablet.cc:2369] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:57:57.352666 16392 raft_consensus.cc:482] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:57:57.352955 16392 raft_consensus.cc:504] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } } peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } }
I20260927 06:57:57.354642 16392 leader_election.cc:288] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66daad4cb9374f83a86b311ad2cf6272 (127.0.97.3:42465), bc2c9c376ab7450fbd7893fe5112ec48 (127.0.97.1:45395)
I20260927 06:57:57.361194 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:57.355202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60063 (local address 127.0.97.3:42465)
0927 06:57:57.355545 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:57.355554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:57.355933 (+ 379us) server_negotiation.cc:407] Connection header received
0927 06:57:57.356180 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:57.356184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:57.356243 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:57.356340 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:57.356891 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.357625 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.359510 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.360265 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.360409 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:57.360529 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:57.360818 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 06:57:57.360943 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260927 06:57:57.361294 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:57.354934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42465 (local address 127.0.97.2:60063)
0927 06:57:57.355806 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:57.355825 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:57.356024 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:57.356361 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:57.356365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:57.356391 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:57.356740 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:57.356751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.357763 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.357770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:57.359350 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:57.359362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.360485 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.360491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:57.360654 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:57.360681 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 06:57:57.360795 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 06:57:57.360934 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":500,"threads_started":1}
I20260927 06:57:57.361421 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:57.355067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59121 (local address 127.0.97.1:45395)
0927 06:57:57.355328 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:57.355336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:57.356192 (+ 856us) server_negotiation.cc:407] Connection header received
0927 06:57:57.356394 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:57.356398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:57.356456 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:57.356564 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:57:57.357124 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.357821 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.359752 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.360492 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.360660 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:57.360761 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:57.361039 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 06:57:57.361164 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 06:57:57.361704 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:57.354940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45395 (local address 127.0.97.2:59121)
0927 06:57:57.356074 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0927 06:57:57.356088 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 06:57:57.356243 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:57:57.356579 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:57:57.356583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:57:57.356605 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:57:57.356963 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:57.356972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.357962 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.357968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:57.359592 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:57:57.359606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.360721 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.360727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:57:57.360893 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:57.360915 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 06:57:57.361015 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 06:57:57.361152 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"mutex_wait_us":56,"thread_start_us":750,"threads_started":1}
I20260927 06:57:57.362365 16321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a7bb949a464d59ba15e41231a5c888" candidate_uuid: "313768f6bc634745b93a16c93271bb36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66daad4cb9374f83a86b311ad2cf6272" is_pre_election: true
I20260927 06:57:57.362560 16054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a7bb949a464d59ba15e41231a5c888" candidate_uuid: "313768f6bc634745b93a16c93271bb36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" is_pre_election: true
I20260927 06:57:57.362874 16321 raft_consensus.cc:2454] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 313768f6bc634745b93a16c93271bb36 in term 0.
I20260927 06:57:57.363066 16054 raft_consensus.cc:2454] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 313768f6bc634745b93a16c93271bb36 in term 0.
I20260927 06:57:57.363592 16122 leader_election.cc:302] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 313768f6bc634745b93a16c93271bb36, 66daad4cb9374f83a86b311ad2cf6272; no voters:
I20260927 06:57:57.364001 16392 raft_consensus.cc:2781] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:57:57.364151 16392 raft_consensus.cc:482] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:57:57.364246 16392 raft_consensus.cc:3037] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:57.366443 16392 raft_consensus.cc:504] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } } peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } }
I20260927 06:57:57.367057 16392 leader_election.cc:288] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [CANDIDATE]: Term 1 election: Requested vote from peers 66daad4cb9374f83a86b311ad2cf6272 (127.0.97.3:42465), bc2c9c376ab7450fbd7893fe5112ec48 (127.0.97.1:45395)
I20260927 06:57:57.367446 16321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a7bb949a464d59ba15e41231a5c888" candidate_uuid: "313768f6bc634745b93a16c93271bb36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66daad4cb9374f83a86b311ad2cf6272"
I20260927 06:57:57.367486 16054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a7bb949a464d59ba15e41231a5c888" candidate_uuid: "313768f6bc634745b93a16c93271bb36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2c9c376ab7450fbd7893fe5112ec48"
I20260927 06:57:57.367638 16321 raft_consensus.cc:3037] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:57.367763 16054 raft_consensus.cc:3037] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:57:57.369853 16054 raft_consensus.cc:2454] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 313768f6bc634745b93a16c93271bb36 in term 1.
I20260927 06:57:57.370288 16124 leader_election.cc:302] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 313768f6bc634745b93a16c93271bb36, bc2c9c376ab7450fbd7893fe5112ec48; no voters:
I20260927 06:57:57.370549 16392 raft_consensus.cc:2781] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:57:57.370962 16321 raft_consensus.cc:2454] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 313768f6bc634745b93a16c93271bb36 in term 1.
I20260927 06:57:57.371001 16392 raft_consensus.cc:686] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [term 1 LEADER]: Becoming Leader. State: Replica: 313768f6bc634745b93a16c93271bb36, State: Running, Role: LEADER
I20260927 06:57:57.371548 16392 consensus_queue.cc:237] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } } peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } }
I20260927 06:57:57.376034 15909 catalog_manager.cc:5495] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 313768f6bc634745b93a16c93271bb36 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "313768f6bc634745b93a16c93271bb36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } health_report { overall_health: HEALTHY } } }
I20260927 06:57:57.445210 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:57.439811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.0.97.2:38027)
0927 06:57:57.440070 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:57.440076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:57.440097 (+ 21us) server_negotiation.cc:407] Connection header received
0927 06:57:57.440203 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:57.440207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:57.440280 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:57.440429 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:57:57.440797 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.441525 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.443152 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.443616 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.443746 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:57.444209 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:57:57.444664 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:57:57.444825 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:57.444875 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 06:57:57.444950 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 06:57:57.453081 16321 raft_consensus.cc:1261] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272 [term 1 FOLLOWER]: Refusing update from remote peer 313768f6bc634745b93a16c93271bb36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:57:57.453094 16054 raft_consensus.cc:1261] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Refusing update from remote peer 313768f6bc634745b93a16c93271bb36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:57:57.454082 16395 consensus_queue.cc:1035] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:57:57.454615 16392 consensus_queue.cc:1035] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:57:57.465287 16398 mvcc.cc:205] Tried to move back new op lower bound from 7333856368438345728 to 7333856368117526528. Current Snapshot: MvccSnapshot[applied={T|T < 7333856368438345728}]
I20260927 06:57:57.478505 15910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:57:57.478788 15910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:57:57.482184 15910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: Sending DeleteTablet for 3 replicas of tablet 68a7bb949a464d59ba15e41231a5c888
I20260927 06:57:57.483326 16301 tablet_service.cc:1514] Processing DeleteTablet for tablet 68a7bb949a464d59ba15e41231a5c888 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:57:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52540
I20260927 06:57:57.483320 16034 tablet_service.cc:1514] Processing DeleteTablet for tablet 68a7bb949a464d59ba15e41231a5c888 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:57:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48752
I20260927 06:57:57.483332 16168 tablet_service.cc:1514] Processing DeleteTablet for tablet 68a7bb949a464d59ba15e41231a5c888 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:57:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38820
I20260927 06:57:57.487053 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 15971
I20260927 06:57:57.496150 16407 tablet_replica.cc:330] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272: stopping tablet replica
I20260927 06:57:57.496891 16408 tablet_replica.cc:330] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36: stopping tablet replica
I20260927 06:57:57.497611 16408 raft_consensus.cc:2229] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:57:57.498195 16407 raft_consensus.cc:2229] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:57:57.498492 16409 tablet_replica.cc:330] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48: stopping tablet replica
I20260927 06:57:57.498788 16407 raft_consensus.cc:2258] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:57:57.499001 16409 raft_consensus.cc:2229] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:57:57.499436 16408 raft_consensus.cc:2258] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:57:57.499502 16409 raft_consensus.cc:2258] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:57:57.501593 16409 ts_tablet_manager.cc:1905] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:57:57.501618 16408 ts_tablet_manager.cc:1905] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:57:57.501624 16407 ts_tablet_manager.cc:1905] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:57:57.508545 16409 ts_tablet_manager.cc:1918] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:57:57.508734 16409 log.cc:1196] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/68a7bb949a464d59ba15e41231a5c888
I20260927 06:57:57.509286 16409 ts_tablet_manager.cc:1939] T 68a7bb949a464d59ba15e41231a5c888 P bc2c9c376ab7450fbd7893fe5112ec48: Deleting consensus metadata
I20260927 06:57:57.509632 16407 ts_tablet_manager.cc:1918] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:57:57.511509 15897 catalog_manager.cc:4841] TS bc2c9c376ab7450fbd7893fe5112ec48 (127.0.97.1:45395): tablet 68a7bb949a464d59ba15e41231a5c888 (table test-table [id=47dbc8802be8445cb4a9c36e1afd1c53]) successfully deleted
I20260927 06:57:57.513895 16407 log.cc:1196] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/68a7bb949a464d59ba15e41231a5c888
I20260927 06:57:57.514469 16407 ts_tablet_manager.cc:1939] T 68a7bb949a464d59ba15e41231a5c888 P 66daad4cb9374f83a86b311ad2cf6272: Deleting consensus metadata
I20260927 06:57:57.516268 15895 catalog_manager.cc:4841] TS 66daad4cb9374f83a86b311ad2cf6272 (127.0.97.3:42465): tablet 68a7bb949a464d59ba15e41231a5c888 (table test-table [id=47dbc8802be8445cb4a9c36e1afd1c53]) successfully deleted
I20260927 06:57:57.518865 16408 ts_tablet_manager.cc:1918] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:57:57.519057 16408 log.cc:1196] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/68a7bb949a464d59ba15e41231a5c888
I20260927 06:57:57.519618 16408 ts_tablet_manager.cc:1939] T 68a7bb949a464d59ba15e41231a5c888 P 313768f6bc634745b93a16c93271bb36: Deleting consensus metadata
I20260927 06:57:57.521452 15896 catalog_manager.cc:4841] TS 313768f6bc634745b93a16c93271bb36 (127.0.97.2:38027): tablet 68a7bb949a464d59ba15e41231a5c888 (table test-table [id=47dbc8802be8445cb4a9c36e1afd1c53]) successfully deleted
Sep 27 06:57:57 dist-test-slave-f892 krb5kdc[15865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:57:57.531169 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:57.488286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48768 (local address 127.0.97.1:45395)
0927 06:57:57.488456 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:57.488462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:57.510943 (+ 22481us) server_negotiation.cc:407] Connection header received
0927 06:57:57.511499 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:57.511505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:57.511585 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:57.511670 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:57.513854 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.515009 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.517618 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.518282 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.518427 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:57.523764 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:57.523778 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:57.523796 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:57.523835 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:57.527841 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:57.528742 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:57.528752 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:57.528756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:57.528908 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:57.529414 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:57.529421 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:57.529424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:57.529859 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:57.530185 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:57.530544 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 06:57:57.530770 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 06:57:57.531884 16094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:57.647507 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 15971
I20260927 06:57:57.665887 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 16105
Sep 27 06:57:57 dist-test-slave-f892 krb5kdc[15865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:57:57.683322 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:57.666696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.0.97.2:38027)
0927 06:57:57.666888 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:57.666893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:57.666911 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:57:57.667101 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:57.667105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:57.667166 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:57.667252 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:57.668945 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.669763 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.671759 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.672177 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.672279 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:57.677532 (+ 5253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:57.677545 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:57.677560 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:57.677600 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:57.680659 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:57.681403 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:57.681411 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:57.681414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:57.681553 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:57.681921 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:57.681924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:57.681926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:57.682320 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:57.682564 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:57.682850 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 06:57:57.683025 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 06:57:57.683981 16228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:57.796393 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 16105
I20260927 06:57:57.815354 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 16238
Sep 27 06:57:57 dist-test-slave-f892 krb5kdc[15865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 06:57:57.829320 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:57:57.816146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52548 (local address 127.0.97.3:42465)
0927 06:57:57.816330 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 06:57:57.816335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:57:57.816353 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:57:57.816514 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:57:57.816519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:57:57.816591 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:57:57.816674 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:57:57.817914 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.818670 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.819871 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:57:57.820235 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:57:57.820322 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:57:57.824492 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:57:57.824505 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:57:57.824521 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:57:57.824551 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:57:57.827040 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:57.827605 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:57.827611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:57.827613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:57.827744 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:57:57.828048 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:57:57.828051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:57:57.828053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:57:57.828362 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:57:57.828568 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 06:57:57.828798 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 06:57:57.828958 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 06:57:57.829943 16361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:57.918306 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 16238
I20260927 06:57:57.936308 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 15878
I20260927 06:57:57.937422 15940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:57:58.048098 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 15878
I20260927 06:58:18.069828 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45499
--webserver_interface=127.0.97.62
--webserver_port=45477
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:58:18.381126 16415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:58:18.381453 16415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:58:18.381615 16415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:58:18.388667 16415 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 06:58:18.388796 16415 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 06:58:18.388888 16415 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 06:58:18.388988 16415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:58:18.389081 16415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:58:18.389200 16415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:58:18.389287 16415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:58:18.389366 16415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:58:18.399039 16415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:58:18.400260 16415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:58:18.401535 16415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:58:18.423859 16422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:58:18.423805 16421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:58:18.423889 16424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:58:18.425019 16415 server_base.cc:1034] running on GCE node
I20260927 06:58:18.425849 16415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:58:18.427353 16415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:58:18.428573 16415 hybrid_clock.cc:648] HybridClock initialized: now 1790492298428510 us; error 33 us; skew 500 ppm
I20260927 06:58:18.428965 16415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:58:18 dist-test-slave-f892 krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:58:18.437284 16415 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:58:18.439040 16415 webserver.cc:460] Webserver started at http://127.0.97.62:45477/ using document root <none> and password file <none>
I20260927 06:58:18.439785 16415 fs_manager.cc:359] Metadata directory not provided
I20260927 06:58:18.439927 16415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:58:18.445771 16415 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 06:58:18.448575 16431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:18.449864 16415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 06:58:18.450098 16415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "ef828650a9aa458aa7e0029f55175129"
format_stamp: "Formatted at 2026-09-27 06:57:55 on dist-test-slave-f892"
server_key: "83061500fde877d5db55a5c936037449"
server_key_iv: "9f3075a826dfe19255dde3560a6c1894"
server_key_version: "encryptionkey@0"
I20260927 06:58:18.451226 16415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:58:18.473376 16415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:58:18.477262 16415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:58:18.477566 16415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:58:18.496064 16415 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45499
I20260927 06:58:18.496097 16482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45499 every 8 connection(s)
I20260927 06:58:18.497529 16415 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:58:18.501657 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 16415
I20260927 06:58:18.502490 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45395
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45499
--webserver_port=37755
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:58:18.504225 16483 sys_catalog.cc:263] Verifying existing consensus state
I20260927 06:58:18.508668 16483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: Bootstrap starting.
I20260927 06:58:18.527844 16483 log.cc:824] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: Log is configured to *not* fsync() on all Append() calls
I20260927 06:58:18.539469 16483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 06:58:18.539994 16483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: Bootstrap complete.
I20260927 06:58:18.548883 16483 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef828650a9aa458aa7e0029f55175129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260927 06:58:18.549769 16483 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef828650a9aa458aa7e0029f55175129, State: Initialized, Role: FOLLOWER
I20260927 06:58:18.550418 16483 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef828650a9aa458aa7e0029f55175129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260927 06:58:18.550688 16483 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:58:18.550830 16483 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:58:18.551014 16483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 1 FOLLOWER]: Advancing to term 2
I20260927 06:58:18.554955 16483 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef828650a9aa458aa7e0029f55175129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260927 06:58:18.555493 16483 leader_election.cc:302] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef828650a9aa458aa7e0029f55175129; no voters:
I20260927 06:58:18.556187 16483 leader_election.cc:288] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 06:58:18.556319 16487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 06:58:18.558415 16487 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 2 LEADER]: Becoming Leader. State: Replica: ef828650a9aa458aa7e0029f55175129, State: Running, Role: LEADER
I20260927 06:58:18.558599 16483 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:58:18.559033 16487 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef828650a9aa458aa7e0029f55175129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260927 06:58:18.563462 16489 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef828650a9aa458aa7e0029f55175129. Latest consensus state: current_term: 2 leader_uuid: "ef828650a9aa458aa7e0029f55175129" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef828650a9aa458aa7e0029f55175129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } }
I20260927 06:58:18.563851 16489 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [sys.catalog]: This master's current role is: LEADER
I20260927 06:58:18.563700 16488 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ef828650a9aa458aa7e0029f55175129" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef828650a9aa458aa7e0029f55175129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } }
I20260927 06:58:18.564107 16488 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [sys.catalog]: This master's current role is: LEADER
I20260927 06:58:18.578577 16497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:58:18.586585 16497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:58:18.588084 16497 catalog_manager.cc:1259] Loaded cluster ID: 4c2bb57af5bf40d98628297bd2f6a74f
I20260927 06:58:18.588227 16497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:58:18.592025 16497 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:58:18.593442 16497 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: Loaded TSK: 0
I20260927 06:58:18.629841 16497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: Generated new TSK 1
I20260927 06:58:18.630330 16497 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 06:58:18.840340 16485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:58:18.840612 16485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:58:18.840759 16485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:58:18.847999 16485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:58:18.848098 16485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:58:18.848260 16485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:58:18.858031 16485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37755
--tserver_master_addrs=127.0.97.62:45499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:58:18.859371 16485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:58:18.860766 16485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:58:18.875749 16514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:58:18.876411 16485 server_base.cc:1034] running on GCE node
W20260927 06:58:18.879160 16512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:58:18.879274 16511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:58:18.879958 16485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:58:18.881789 16485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:58:18.883081 16485 hybrid_clock.cc:648] HybridClock initialized: now 1790492298882996 us; error 80 us; skew 500 ppm
I20260927 06:58:18.883474 16485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:58:18 dist-test-slave-f892 krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:58:18.886948 16485 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:58:18.888526 16485 webserver.cc:460] Webserver started at http://127.0.97.1:37755/ using document root <none> and password file <none>
I20260927 06:58:18.889189 16485 fs_manager.cc:359] Metadata directory not provided
I20260927 06:58:18.889350 16485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:58:18.894589 16485 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:58:18.897208 16521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:18.898634 16485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:58:18.898794 16485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "bc2c9c376ab7450fbd7893fe5112ec48"
format_stamp: "Formatted at 2026-09-27 06:57:55 on dist-test-slave-f892"
server_key: "25459d9c65c7f2354730f04a0b216abf"
server_key_iv: "0af4105c19f416eccdde695320cf6d8e"
server_key_version: "encryptionkey@0"
I20260927 06:58:18.899730 16485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:58:18.933506 16485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:58:18.937253 16485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:58:18.937476 16485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:58:18.938381 16485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:58:18.939563 16485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:58:18.939646 16485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:18.939723 16485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:58:18.939774 16485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:18.971504 16485 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45395
I20260927 06:58:18.971588 16633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45395 every 8 connection(s)
I20260927 06:58:18.972869 16485 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:58:18.974658 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 16485
I20260927 06:58:18.975324 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38027
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45499
--webserver_port=40523
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 06:58:18 dist-test-slave-f892 krb5kdc[15865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:58:19.000468 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:18.975709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55345 (local address 127.0.97.62:45499)
0927 06:58:18.981660 (+ 5951us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:18.981695 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:18.981743 (+ 48us) server_negotiation.cc:407] Connection header received
0927 06:58:18.982125 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:18.982144 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:18.982550 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:18.982921 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:58:18.984709 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:18.986728 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:18.988217 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:18.989296 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:18.989472 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:18.993237 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:58:18.993244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:58:18.993256 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:58:18.993287 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:58:18.996135 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:18.997080 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:18.997086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:18.997090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:18.997233 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:18.997722 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:18.997725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:18.997727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:18.998241 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:58:18.998373 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:18.999363 (+ 990us) server_negotiation.cc:299] Negotiation successful
0927 06:58:18.999648 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5717,"thread_start_us":5606,"threads_started":1}
I20260927 06:58:19.001027 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:18.975860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:55345)
0927 06:58:18.976856 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:18.976909 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 06:58:18.979505 (+ 2596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:58:18.983143 (+ 3638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:58:18.983155 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:58:18.983909 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:58:18.984521 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:18.984538 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:18.986896 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:18.986914 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:18.988063 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:18.988074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:18.989484 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:18.989488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:18.989653 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:18.990326 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:58:18.990350 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:58:18.993002 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:58:18.996284 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:58:18.996295 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:58:18.996312 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:58:18.996958 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:58:18.997355 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:58:18.997363 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:58:18.997368 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:58:18.997607 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:58:18.998393 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:58:18.998402 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:58:18.998839 (+ 437us) client_negotiation.cc:769] Sending connection context
0927 06:58:18.999178 (+ 339us) client_negotiation.cc:240] Negotiation successful
0927 06:58:18.999729 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":379,"threads_started":1}
I20260927 06:58:19.003824 16634 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45499
I20260927 06:58:19.004060 16634 heartbeater.cc:464] Registering TS with master...
I20260927 06:58:19.004840 16634 heartbeater.cc:511] Master 127.0.97.62:45499 requested a full tablet report, sending...
I20260927 06:58:19.007117 16448 ts_manager.cc:194] Registered new tserver with Master: bc2c9c376ab7450fbd7893fe5112ec48 (127.0.97.1:45395)
I20260927 06:58:19.009208 16448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55345
W20260927 06:58:19.293074 16638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:58:19.293342 16638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:58:19.293435 16638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:58:19.300498 16638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:58:19.300604 16638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:58:19.300717 16638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:58:19.310928 16638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40523
--tserver_master_addrs=127.0.97.62:45499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:58:19.312100 16638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:58:19.313318 16638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:58:19.326223 16644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:58:19.326337 16643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:58:19.326309 16646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:58:19.327049 16638 server_base.cc:1034] running on GCE node
I20260927 06:58:19.327678 16638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:58:19.328598 16638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:58:19.329838 16638 hybrid_clock.cc:648] HybridClock initialized: now 1790492299329792 us; error 55 us; skew 500 ppm
I20260927 06:58:19.330235 16638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:58:19 dist-test-slave-f892 krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:58:19.333186 16638 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:58:19.334580 16638 webserver.cc:460] Webserver started at http://127.0.97.2:40523/ using document root <none> and password file <none>
I20260927 06:58:19.335161 16638 fs_manager.cc:359] Metadata directory not provided
I20260927 06:58:19.335265 16638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:58:19.340181 16638 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 06:58:19.342851 16653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:19.344074 16638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 06:58:19.344197 16638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "313768f6bc634745b93a16c93271bb36"
format_stamp: "Formatted at 2026-09-27 06:57:56 on dist-test-slave-f892"
server_key: "cfa03080971567a9aa0ad46b37b8dca8"
server_key_iv: "4c6913969f997a7addb7e699620f6a91"
server_key_version: "encryptionkey@0"
I20260927 06:58:19.345063 16638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:58:19.368189 16638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:58:19.372219 16638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:58:19.372437 16638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:58:19.373288 16638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:58:19.374483 16638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:58:19.374567 16638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:19.374645 16638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:58:19.374697 16638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:19.407143 16638 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38027
I20260927 06:58:19.407231 16765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38027 every 8 connection(s)
I20260927 06:58:19.408555 16638 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:58:19.410624 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 16638
I20260927 06:58:19.411357 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42465
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45499
--webserver_port=38371
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 06:58:19 dist-test-slave-f892 krb5kdc[15865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:58:19.428275 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:19.411135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42177 (local address 127.0.97.62:45499)
0927 06:58:19.411370 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:19.411379 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:19.412615 (+ 1236us) server_negotiation.cc:407] Connection header received
0927 06:58:19.414601 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:19.414609 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:19.414709 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:19.414824 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:58:19.416321 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.417025 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.418659 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.419304 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.419454 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:19.422839 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:58:19.422844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:58:19.422847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:58:19.422880 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:58:19.425332 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:19.426043 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:19.426051 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:19.426057 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:19.426186 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:19.426604 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:19.426609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:19.426612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:19.426886 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:58:19.427045 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:19.427698 (+ 653us) server_negotiation.cc:299] Negotiation successful
0927 06:58:19.427940 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 06:58:19.429275 16768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:19.411653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:42177)
0927 06:58:19.412450 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:19.412501 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 06:58:19.414326 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:58:19.415071 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:58:19.415079 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:58:19.415578 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:58:19.416139 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:19.416156 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.417201 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.417206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:19.418502 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:19.418514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.419582 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.419587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:19.419784 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:19.420476 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:58:19.420498 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:58:19.422606 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:58:19.425472 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:58:19.425480 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:58:19.425494 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:58:19.425894 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:58:19.426292 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:58:19.426298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:58:19.426302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:58:19.426486 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:58:19.427028 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:58:19.427035 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:58:19.427346 (+ 311us) client_negotiation.cc:769] Sending connection context
0927 06:58:19.427605 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 06:58:19.428102 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":340,"threads_started":1}
I20260927 06:58:19.430799 16766 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45499
I20260927 06:58:19.431115 16766 heartbeater.cc:464] Registering TS with master...
I20260927 06:58:19.431799 16766 heartbeater.cc:511] Master 127.0.97.62:45499 requested a full tablet report, sending...
I20260927 06:58:19.433197 16448 ts_manager.cc:194] Registered new tserver with Master: 313768f6bc634745b93a16c93271bb36 (127.0.97.2:38027)
I20260927 06:58:19.434141 16448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42177
W20260927 06:58:19.732175 16769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:58:19.732452 16769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:58:19.732739 16769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:58:19.740209 16769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:58:19.740310 16769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:58:19.740471 16769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:58:19.750298 16769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38371
--tserver_master_addrs=127.0.97.62:45499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:58:19.751531 16769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:58:19.752936 16769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:58:19.766217 16775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:58:19.766247 16777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:58:19.766304 16774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:58:19.767199 16769 server_base.cc:1034] running on GCE node
I20260927 06:58:19.767788 16769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:58:19.768651 16769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:58:19.769871 16769 hybrid_clock.cc:648] HybridClock initialized: now 1790492299769837 us; error 47 us; skew 500 ppm
I20260927 06:58:19.770215 16769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:58:19 dist-test-slave-f892 krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:58:19.773090 16769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:58:19.774528 16769 webserver.cc:460] Webserver started at http://127.0.97.3:38371/ using document root <none> and password file <none>
I20260927 06:58:19.775113 16769 fs_manager.cc:359] Metadata directory not provided
I20260927 06:58:19.775223 16769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:58:19.780119 16769 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 06:58:19.782580 16784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:19.783787 16769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 06:58:19.783943 16769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "66daad4cb9374f83a86b311ad2cf6272"
format_stamp: "Formatted at 2026-09-27 06:57:56 on dist-test-slave-f892"
server_key: "93f3078ed97acb1930c71cfbf83b00ab"
server_key_iv: "4f590b0ec4fe35629c42e4b069a71025"
server_key_version: "encryptionkey@0"
I20260927 06:58:19.784830 16769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:58:19.820775 16769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:58:19.824559 16769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:58:19.824764 16769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:58:19.825639 16769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:58:19.826895 16769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:58:19.826990 16769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:19.827067 16769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:58:19.827119 16769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:19.859253 16769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42465
I20260927 06:58:19.859378 16896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42465 every 8 connection(s)
I20260927 06:58:19.860718 16769 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:58:19.865347 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 16769
Sep 27 06:58:19 dist-test-slave-f892 krb5kdc[15865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:58:19.880894 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:19.866274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50506 (local address 127.0.97.62:45499)
0927 06:58:19.866987 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:19.866996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:19.867515 (+ 519us) server_negotiation.cc:407] Connection header received
0927 06:58:19.867627 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:19.867632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:19.867703 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:19.867815 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:58:19.869249 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.869984 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.871745 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.872189 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.872340 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:19.875196 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:58:19.875200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:58:19.875204 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:58:19.875232 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:58:19.878382 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:19.878980 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:19.878987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:19.878990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:19.879099 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:19.879459 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:19.879466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:19.879469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:19.879712 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:58:19.879982 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:19.880285 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 06:58:19.880479 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":397,"threads_started":1}
I20260927 06:58:19.881237 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:19.863143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60283 (local address 127.0.97.62:45499)
0927 06:58:19.863330 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:19.863334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:19.864433 (+ 1099us) server_negotiation.cc:407] Connection header received
0927 06:58:19.866452 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:19.866461 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:19.866574 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:19.866726 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:58:19.868457 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.869183 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.871175 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.871587 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.871703 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:19.875445 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:58:19.875450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:58:19.875453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:58:19.875496 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:58:19.878439 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:19.879130 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:19.879136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:19.879138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:19.879241 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:19.879648 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:19.879652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:19.879657 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:19.879919 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:58:19.880076 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:19.880716 (+ 640us) server_negotiation.cc:299] Negotiation successful
0927 06:58:19.880909 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 06:58:19.882336 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:19.863580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:60283)
0927 06:58:19.864276 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:19.864315 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 06:58:19.866093 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:58:19.867073 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:58:19.867080 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:58:19.867662 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:58:19.868289 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:19.868306 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.869338 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.869344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:19.871013 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:19.871028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.871799 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.871804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:19.871990 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:19.872685 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:58:19.872716 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:58:19.875244 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:58:19.878575 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:58:19.878583 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:58:19.878603 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:58:19.878998 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:58:19.879350 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:58:19.879355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:58:19.879359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:58:19.879529 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:58:19.880057 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:58:19.880064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:58:19.880377 (+ 313us) client_negotiation.cc:769] Sending connection context
0927 06:58:19.880612 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 06:58:19.881074 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":279,"threads_started":1}
I20260927 06:58:19.883674 16897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45499
I20260927 06:58:19.883965 16897 heartbeater.cc:464] Registering TS with master...
I20260927 06:58:19.884605 16897 heartbeater.cc:511] Master 127.0.97.62:45499 requested a full tablet report, sending...
I20260927 06:58:19.885845 16448 ts_manager.cc:194] Registered new tserver with Master: 66daad4cb9374f83a86b311ad2cf6272 (127.0.97.3:42465)
I20260927 06:58:19.886657 16448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60283
I20260927 06:58:19.895677 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:58:19.901543 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:19.896483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50512 (local address 127.0.97.62:45499)
0927 06:58:19.896678 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:19.896687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:19.897040 (+ 353us) server_negotiation.cc:407] Connection header received
0927 06:58:19.897188 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:19.897191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:19.897304 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:19.897426 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:58:19.897793 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.898583 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.900034 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.900415 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.900503 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:19.900906 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:58:19.901068 (+ 162us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 06:58:19.901316 (+ 248us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50512: token expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 06:58:19.901659 16636 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50512: token expired
W20260927 06:58:19.902190 16372 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:45499)
I20260927 06:58:19.913798 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:19.902682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50522 (local address 127.0.97.62:45499)
0927 06:58:19.902839 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:19.902846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:19.902863 (+ 17us) server_negotiation.cc:407] Connection header received
0927 06:58:19.902906 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:19.902910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:19.902967 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:19.903089 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:58:19.904454 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.905193 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.906490 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.906854 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.906945 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:19.909304 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:58:19.909308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:58:19.909311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:58:19.909335 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:58:19.911624 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:19.912205 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:19.912209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:19.912211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:19.912314 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:19.912614 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:19.912617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:19.912619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:19.912886 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:58:19.913061 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:19.913300 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 06:58:19.913455 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 06:58:19.916834 16448 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:58:19.923753 16448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:58:19.945708 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:19.934583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38027 (local address 127.0.0.1:33124)
0927 06:58:19.935509 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:19.935555 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 06:58:19.935710 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:58:19.938722 (+ 3012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:58:19.938725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:58:19.938744 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:58:19.939141 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:19.939151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.941048 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.941055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:19.943157 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:19.943168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.944719 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.944725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:19.944921 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:19.944968 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 06:58:19.945091 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 06:58:19.945269 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":63,"thread_start_us":1292,"threads_started":1}
I20260927 06:58:19.946161 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:19.934820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42465 (local address 127.0.0.1:32798)
0927 06:58:19.936788 (+ 1968us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:19.936804 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:58:19.936917 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:58:19.938473 (+ 1556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:58:19.938482 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:58:19.938524 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:58:19.938932 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:19.938945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.940885 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.940890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:19.943535 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:19.943550 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.945162 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.945167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:19.945366 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:19.945388 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 06:58:19.945493 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 06:58:19.945609 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1852,"mutex_wait_us":74,"thread_start_us":3059,"threads_started":1}
I20260927 06:58:19.946467 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:19.934654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.0.97.2:38027)
0927 06:58:19.937806 (+ 3152us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:19.937814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:19.937840 (+ 26us) server_negotiation.cc:407] Connection header received
0927 06:58:19.937943 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:19.937951 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:19.938221 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:19.938450 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:58:19.939318 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.940898 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.943300 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.944526 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.944645 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:19.944830 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:19.945943 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0927 06:58:19.946128 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3002,"thread_start_us":2870,"threads_started":1}
I20260927 06:58:19.946506 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:19.935507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32798 (local address 127.0.97.3:42465)
0927 06:58:19.937899 (+ 2392us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:19.937906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:19.937931 (+ 25us) server_negotiation.cc:407] Connection header received
0927 06:58:19.938025 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:19.938032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:19.938220 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:19.938450 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:58:19.939158 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.940746 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.943702 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.944960 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.945099 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:19.945308 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:19.945540 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 06:58:19.945681 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2244,"thread_start_us":2073,"threads_started":1}
I20260927 06:58:19.946842 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:19.935782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35116 (local address 127.0.97.1:45395)
0927 06:58:19.937378 (+ 1596us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:19.937387 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:19.938322 (+ 935us) server_negotiation.cc:407] Connection header received
0927 06:58:19.938570 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:19.938577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:19.938796 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:19.938951 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:58:19.939752 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.941146 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.943996 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.945313 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.945411 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:19.945581 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:19.946387 (+ 806us) server_negotiation.cc:299] Negotiation successful
0927 06:58:19.946552 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1449,"thread_start_us":1434,"threads_started":1}
I20260927 06:58:19.947486 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:19.935211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45395 (local address 127.0.0.1:35116)
0927 06:58:19.938206 (+ 2995us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:19.938220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 06:58:19.938355 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:58:19.939213 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:58:19.939216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:58:19.939258 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:58:19.939585 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:19.939597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.941274 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.941279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:19.943682 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:19.943693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:19.945981 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:19.945987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:19.946181 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:19.946215 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 06:58:19.947020 (+ 805us) client_negotiation.cc:240] Negotiation successful
0927 06:58:19.947147 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2834,"mutex_wait_us":44,"thread_start_us":1321,"threads_started":1}
I20260927 06:58:19.950062 16832 tablet_service.cc:1467] Processing CreateTablet for tablet 538ca8ed8e2b430187068ec14ba5a9c0 (DEFAULT_TABLE table=test-table [id=5ec70daca36344fa88b0986038e74c12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:58:19.950747 16569 tablet_service.cc:1467] Processing CreateTablet for tablet 538ca8ed8e2b430187068ec14ba5a9c0 (DEFAULT_TABLE table=test-table [id=5ec70daca36344fa88b0986038e74c12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:58:19.951548 16832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 538ca8ed8e2b430187068ec14ba5a9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:58:19.952018 16569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 538ca8ed8e2b430187068ec14ba5a9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:58:19.951967 16701 tablet_service.cc:1467] Processing CreateTablet for tablet 538ca8ed8e2b430187068ec14ba5a9c0 (DEFAULT_TABLE table=test-table [id=5ec70daca36344fa88b0986038e74c12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:58:19.952987 16701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 538ca8ed8e2b430187068ec14ba5a9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:58:19.964054 16913 tablet_bootstrap.cc:492] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36: Bootstrap starting.
I20260927 06:58:19.966099 16914 tablet_bootstrap.cc:492] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272: Bootstrap starting.
I20260927 06:58:19.968066 16913 tablet_bootstrap.cc:654] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36: Neither blocks nor log segments found. Creating new log.
I20260927 06:58:19.969323 16913 log.cc:824] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36: Log is configured to *not* fsync() on all Append() calls
I20260927 06:58:19.969415 16914 tablet_bootstrap.cc:654] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272: Neither blocks nor log segments found. Creating new log.
I20260927 06:58:19.972517 16913 tablet_bootstrap.cc:492] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36: No bootstrap required, opened a new log
I20260927 06:58:19.972749 16913 ts_tablet_manager.cc:1397] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 06:58:19.970655 16914 log.cc:824] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272: Log is configured to *not* fsync() on all Append() calls
I20260927 06:58:19.974200 16915 tablet_bootstrap.cc:492] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48: Bootstrap starting.
I20260927 06:58:19.978698 16915 tablet_bootstrap.cc:654] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48: Neither blocks nor log segments found. Creating new log.
I20260927 06:58:19.980221 16915 log.cc:824] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48: Log is configured to *not* fsync() on all Append() calls
I20260927 06:58:19.982568 16913 raft_consensus.cc:348] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } } peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } }
I20260927 06:58:19.983060 16913 raft_consensus.cc:374] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:58:19.983189 16913 raft_consensus.cc:729] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313768f6bc634745b93a16c93271bb36, State: Initialized, Role: FOLLOWER
I20260927 06:58:19.983870 16913 consensus_queue.cc:260] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } } peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } }
I20260927 06:58:19.984272 16914 tablet_bootstrap.cc:492] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272: No bootstrap required, opened a new log
I20260927 06:58:19.984601 16914 ts_tablet_manager.cc:1397] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.004s
I20260927 06:58:19.985484 16913 ts_tablet_manager.cc:1428] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260927 06:58:19.986284 16766 heartbeater.cc:503] Master 127.0.97.62:45499 was elected leader, sending a full tablet report...
I20260927 06:58:19.986495 16915 tablet_bootstrap.cc:492] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48: No bootstrap required, opened a new log
I20260927 06:58:19.986728 16915 ts_tablet_manager.cc:1397] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
I20260927 06:58:19.993855 16914 raft_consensus.cc:348] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } } peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } }
I20260927 06:58:19.994314 16914 raft_consensus.cc:374] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:58:19.994428 16914 raft_consensus.cc:729] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66daad4cb9374f83a86b311ad2cf6272, State: Initialized, Role: FOLLOWER
I20260927 06:58:19.994433 16915 raft_consensus.cc:348] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } } peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } }
I20260927 06:58:19.994823 16915 raft_consensus.cc:374] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:58:19.994925 16915 raft_consensus.cc:729] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc2c9c376ab7450fbd7893fe5112ec48, State: Initialized, Role: FOLLOWER
I20260927 06:58:19.995050 16914 consensus_queue.cc:260] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } } peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } }
I20260927 06:58:19.995419 16915 consensus_queue.cc:260] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } } peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } }
I20260927 06:58:19.996073 16897 heartbeater.cc:503] Master 127.0.97.62:45499 was elected leader, sending a full tablet report...
I20260927 06:58:19.996707 16914 ts_tablet_manager.cc:1428] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260927 06:58:19.997488 16634 heartbeater.cc:503] Master 127.0.97.62:45499 was elected leader, sending a full tablet report...
I20260927 06:58:19.998077 16915 ts_tablet_manager.cc:1428] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48: Time spent starting tablet: real 0.011s user 0.010s sys 0.002s
W20260927 06:58:20.112548 16898 tablet.cc:2369] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:58:20.160517 16767 tablet.cc:2369] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:58:20.225160 16635 tablet.cc:2369] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:58:20.229422 16921 raft_consensus.cc:482] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:58:20.229688 16921 raft_consensus.cc:504] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } } peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } }
I20260927 06:58:20.231441 16921 leader_election.cc:288] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 313768f6bc634745b93a16c93271bb36 (127.0.97.2:38027), 66daad4cb9374f83a86b311ad2cf6272 (127.0.97.3:42465)
I20260927 06:58:20.240621 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:20.231922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34989 (local address 127.0.97.3:42465)
0927 06:58:20.232167 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:20.232176 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:20.233022 (+ 846us) server_negotiation.cc:407] Connection header received
0927 06:58:20.233117 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:20.233121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:20.233211 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:20.233315 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:58:20.234109 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:20.235224 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:20.238324 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:20.239552 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:20.239684 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:20.239783 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:20.240226 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 06:58:20.240357 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 06:58:20.240620 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:20.231783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37683 (local address 127.0.97.2:38027)
0927 06:58:20.232000 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:20.232006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:20.232858 (+ 852us) server_negotiation.cc:407] Connection header received
0927 06:58:20.233069 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:20.233074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:20.233168 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:20.233266 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:58:20.234109 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:20.235234 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:20.238324 (+ 3090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:20.239551 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:20.239684 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:20.239779 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:20.240226 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 06:58:20.240356 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 06:58:20.240908 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:20.231875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42465 (local address 127.0.97.1:34989)
0927 06:58:20.232900 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:20.232914 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 06:58:20.233019 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:58:20.233327 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:58:20.233330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:58:20.233423 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:58:20.233938 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:20.233947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:20.235357 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:20.235362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:20.238152 (+ 2790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:20.238168 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:20.239822 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:20.239826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:20.240018 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:20.240074 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 06:58:20.240188 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 06:58:20.240421 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":324,"spinlock_wait_cycles":19200,"thread_start_us":688,"threads_started":1}
I20260927 06:58:20.240929 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:20.231710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38027 (local address 127.0.97.1:37683)
0927 06:58:20.232738 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:20.232764 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 06:58:20.232921 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:58:20.233310 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:58:20.233314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:58:20.233341 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:58:20.233910 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:20.233928 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:20.235358 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:20.235362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:20.238152 (+ 2790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:20.238168 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:20.239806 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:20.239814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:20.240023 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:20.240076 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 06:58:20.240188 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 06:58:20.240421 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":331,"spinlock_wait_cycles":17280,"thread_start_us":465,"threads_started":1}
I20260927 06:58:20.241922 16852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538ca8ed8e2b430187068ec14ba5a9c0" candidate_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66daad4cb9374f83a86b311ad2cf6272" is_pre_election: true
I20260927 06:58:20.241977 16721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538ca8ed8e2b430187068ec14ba5a9c0" candidate_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313768f6bc634745b93a16c93271bb36" is_pre_election: true
I20260927 06:58:20.242493 16852 raft_consensus.cc:2454] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc2c9c376ab7450fbd7893fe5112ec48 in term 0.
I20260927 06:58:20.242502 16721 raft_consensus.cc:2454] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc2c9c376ab7450fbd7893fe5112ec48 in term 0.
I20260927 06:58:20.243172 16523 leader_election.cc:302] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 313768f6bc634745b93a16c93271bb36, 66daad4cb9374f83a86b311ad2cf6272, bc2c9c376ab7450fbd7893fe5112ec48; no voters:
I20260927 06:58:20.243613 16921 raft_consensus.cc:2781] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:58:20.243757 16921 raft_consensus.cc:482] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:58:20.243875 16921 raft_consensus.cc:3037] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:58:20.246052 16921 raft_consensus.cc:504] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } } peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } }
I20260927 06:58:20.246734 16921 leader_election.cc:288] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 1 election: Requested vote from peers 313768f6bc634745b93a16c93271bb36 (127.0.97.2:38027), 66daad4cb9374f83a86b311ad2cf6272 (127.0.97.3:42465)
I20260927 06:58:20.247179 16721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538ca8ed8e2b430187068ec14ba5a9c0" candidate_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313768f6bc634745b93a16c93271bb36"
I20260927 06:58:20.247258 16852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538ca8ed8e2b430187068ec14ba5a9c0" candidate_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66daad4cb9374f83a86b311ad2cf6272"
I20260927 06:58:20.247392 16721 raft_consensus.cc:3037] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:58:20.247442 16852 raft_consensus.cc:3037] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:58:20.250568 16721 raft_consensus.cc:2454] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc2c9c376ab7450fbd7893fe5112ec48 in term 1.
I20260927 06:58:20.250568 16852 raft_consensus.cc:2454] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc2c9c376ab7450fbd7893fe5112ec48 in term 1.
I20260927 06:58:20.250927 16523 leader_election.cc:302] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66daad4cb9374f83a86b311ad2cf6272, bc2c9c376ab7450fbd7893fe5112ec48; no voters:
I20260927 06:58:20.251286 16921 raft_consensus.cc:2781] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:58:20.251654 16921 raft_consensus.cc:686] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 LEADER]: Becoming Leader. State: Replica: bc2c9c376ab7450fbd7893fe5112ec48, State: Running, Role: LEADER
I20260927 06:58:20.252094 16921 consensus_queue.cc:237] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } } peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } }
I20260927 06:58:20.256407 16447 catalog_manager.cc:5495] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc2c9c376ab7450fbd7893fe5112ec48 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } health_report { overall_health: HEALTHY } } }
I20260927 06:58:20.311087 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:20.305770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.0.97.1:45395)
0927 06:58:20.306003 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:20.306009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:20.306068 (+ 59us) server_negotiation.cc:407] Connection header received
0927 06:58:20.306290 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:20.306294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:20.306371 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:20.306520 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:58:20.306921 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:20.307599 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:20.308914 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:20.309333 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:20.309484 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:20.309942 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:58:20.310479 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:58:20.310637 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:20.310716 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 06:58:20.310794 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 06:58:20.318769 16852 raft_consensus.cc:1261] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272 [term 1 FOLLOWER]: Refusing update from remote peer bc2c9c376ab7450fbd7893fe5112ec48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:58:20.318771 16721 raft_consensus.cc:1261] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36 [term 1 FOLLOWER]: Refusing update from remote peer bc2c9c376ab7450fbd7893fe5112ec48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:58:20.319797 16921 consensus_queue.cc:1035] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:58:20.320111 16924 consensus_queue.cc:1035] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:58:20.344761 16447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:58:20.345041 16447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:58:20.348531 16447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: Sending DeleteTablet for 3 replicas of tablet 538ca8ed8e2b430187068ec14ba5a9c0
I20260927 06:58:20.349650 16701 tablet_service.cc:1514] Processing DeleteTablet for tablet 538ca8ed8e2b430187068ec14ba5a9c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:58:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33124
I20260927 06:58:20.349727 16569 tablet_service.cc:1514] Processing DeleteTablet for tablet 538ca8ed8e2b430187068ec14ba5a9c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:58:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35116
I20260927 06:58:20.349745 16832 tablet_service.cc:1514] Processing DeleteTablet for tablet 538ca8ed8e2b430187068ec14ba5a9c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:58:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32798
I20260927 06:58:20.352154 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 16485
I20260927 06:58:20.369100 16939 tablet_replica.cc:330] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272: stopping tablet replica
I20260927 06:58:20.369983 16939 raft_consensus.cc:2229] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:58:20.370563 16939 raft_consensus.cc:2258] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:58:20.372288 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:20.353057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35136 (local address 127.0.97.1:45395)
0927 06:58:20.353261 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:20.353269 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:20.353293 (+ 24us) server_negotiation.cc:407] Connection header received
0927 06:58:20.353419 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:20.353426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:20.353521 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:20.353650 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:58:20.355365 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:20.356910 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:20.358842 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:20.359850 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:20.360019 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:20.363182 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:58:20.363198 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:58:20.363220 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:58:20.363272 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:58:20.368215 (+ 4943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:20.369024 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:20.369037 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:20.369043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:20.369306 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:20.369723 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:20.369730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:20.369736 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:20.370748 (+ 1012us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:58:20.371147 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:20.371421 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 06:58:20.371731 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 06:58:20.373402 16629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:58:20.373907 16939 ts_tablet_manager.cc:1905] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:58:20.374050 16937 tablet_replica.cc:330] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36: stopping tablet replica
I20260927 06:58:20.374428 16937 raft_consensus.cc:2229] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:58:20.374913 16937 raft_consensus.cc:2258] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:58:20.376756 16937 ts_tablet_manager.cc:1905] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:58:20.380810 16939 ts_tablet_manager.cc:1918] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:58:20.381162 16939 log.cc:1196] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/538ca8ed8e2b430187068ec14ba5a9c0
I20260927 06:58:20.381757 16939 ts_tablet_manager.cc:1939] T 538ca8ed8e2b430187068ec14ba5a9c0 P 66daad4cb9374f83a86b311ad2cf6272: Deleting consensus metadata
I20260927 06:58:20.383718 16937 ts_tablet_manager.cc:1918] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:58:20.383920 16937 log.cc:1196] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/538ca8ed8e2b430187068ec14ba5a9c0
I20260927 06:58:20.384040 16433 catalog_manager.cc:4841] TS 66daad4cb9374f83a86b311ad2cf6272 (127.0.97.3:42465): tablet 538ca8ed8e2b430187068ec14ba5a9c0 (table test-table [id=5ec70daca36344fa88b0986038e74c12]) successfully deleted
I20260927 06:58:20.384407 16937 ts_tablet_manager.cc:1939] T 538ca8ed8e2b430187068ec14ba5a9c0 P 313768f6bc634745b93a16c93271bb36: Deleting consensus metadata
I20260927 06:58:20.386051 16434 catalog_manager.cc:4841] TS 313768f6bc634745b93a16c93271bb36 (127.0.97.2:38027): tablet 538ca8ed8e2b430187068ec14ba5a9c0 (table test-table [id=5ec70daca36344fa88b0986038e74c12]) successfully deleted
I20260927 06:58:20.479745 16938 tablet_replica.cc:330] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48: stopping tablet replica
I20260927 06:58:20.480005 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 16485
W20260927 06:58:20.498793 16435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45395 (error 108)
I20260927 06:58:20.498930 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 16638
I20260927 06:58:20.501109 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:20.500195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45395 (local address 127.0.0.1:35150)
0927 06:58:20.500763 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:20.500878 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":362,"threads_started":1}
W20260927 06:58:20.501361 16435 catalog_manager.cc:4584] TS bc2c9c376ab7450fbd7893fe5112ec48 (127.0.97.1:45395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 538ca8ed8e2b430187068ec14ba5a9c0: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45395: connect: Connection refused (error 111)
I20260927 06:58:20.513669 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:20.499655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33134 (local address 127.0.97.2:38027)
0927 06:58:20.499817 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:20.499822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:20.499976 (+ 154us) server_negotiation.cc:407] Connection header received
0927 06:58:20.500171 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:20.500176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:20.500262 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:20.500359 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:58:20.501926 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:20.502982 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:20.504235 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:20.504913 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:20.505045 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:20.507707 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:58:20.507725 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:58:20.507744 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:58:20.507787 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:58:20.511108 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:20.511741 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:20.511750 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:20.511753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:20.511914 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:20.512242 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:20.512247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:20.512250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:20.512657 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:58:20.512931 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:20.513120 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 06:58:20.513327 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 06:58:20.514350 16761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:58:20.554955 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:20.554450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45395 (local address 127.0.0.1:35166)
0927 06:58:20.554675 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:20.554810 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 06:58:20.555994 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:20.555625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45395 (local address 127.0.0.1:35176)
0927 06:58:20.555810 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:20.555873 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 06:58:20.600994 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 16638
I20260927 06:58:20.616250 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:20.615782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45395 (local address 127.0.0.1:35178)
0927 06:58:20.615978 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:20.616106 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 06:58:20.617377 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:20.616988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45395 (local address 127.0.0.1:35192)
0927 06:58:20.617138 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:20.617201 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 06:58:20.617900 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 16769
I20260927 06:58:20.630425 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:20.618598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32810 (local address 127.0.97.3:42465)
0927 06:58:20.618750 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:20.618754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:20.618772 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:58:20.618929 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:20.618933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:20.618997 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:20.619079 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:58:20.620356 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:20.621114 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:20.622754 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:20.623118 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:20.623209 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:20.625682 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:58:20.625694 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:58:20.625708 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:58:20.625737 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:58:20.628145 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:20.628733 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:20.628740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:20.628742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:20.628873 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:20.629192 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:20.629195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:20.629198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:20.629521 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:58:20.629710 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:20.629951 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 06:58:20.630193 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 06:58:20.630995 16892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:58:20.697933 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:20.697466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45395 (local address 127.0.0.1:35208)
0927 06:58:20.697688 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:20.697777 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 06:58:20.699056 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:20.698628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45395 (local address 127.0.0.1:35220)
0927 06:58:20.698847 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:20.698925 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 06:58:20.718492 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 16769
I20260927 06:58:20.735091 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 16415
I20260927 06:58:20.736189 16478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:58:20.848937 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 16415
I20260927 06:58:40.876597 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45499
--webserver_interface=127.0.97.62
--webserver_port=45477
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:58:41.244445 16946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:58:41.244742 16946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:58:41.244899 16946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:58:41.252236 16946 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 06:58:41.252336 16946 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 06:58:41.252375 16946 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 06:58:41.252475 16946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:58:41.252545 16946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:58:41.252638 16946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:58:41.252723 16946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:58:41.252822 16946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:58:41.269095 16946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:58:41.270536 16946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:58:41.271942 16946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:58:41.284622 16951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:58:41.284592 16952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:58:41.284569 16954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:58:41.285853 16946 server_base.cc:1034] running on GCE node
I20260927 06:58:41.286700 16946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:58:41.288141 16946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:58:41.289391 16946 hybrid_clock.cc:648] HybridClock initialized: now 1790492321289310 us; error 54 us; skew 500 ppm
I20260927 06:58:41.289718 16946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:58:41 dist-test-slave-f892 krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:58:41.292814 16946 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:58:41.294262 16946 webserver.cc:460] Webserver started at http://127.0.97.62:45477/ using document root <none> and password file <none>
I20260927 06:58:41.294934 16946 fs_manager.cc:359] Metadata directory not provided
I20260927 06:58:41.295070 16946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:58:41.300199 16946 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 06:58:41.302877 16961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:41.304284 16946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 06:58:41.304461 16946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "ef828650a9aa458aa7e0029f55175129"
format_stamp: "Formatted at 2026-09-27 06:57:55 on dist-test-slave-f892"
server_key: "83061500fde877d5db55a5c936037449"
server_key_iv: "9f3075a826dfe19255dde3560a6c1894"
server_key_version: "encryptionkey@0"
I20260927 06:58:41.305410 16946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:58:41.319444 16946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:58:41.323191 16946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:58:41.323465 16946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:58:41.341022 16946 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45499
I20260927 06:58:41.341070 17012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45499 every 8 connection(s)
I20260927 06:58:41.342470 16946 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:58:41.343259 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 16946
I20260927 06:58:41.344197 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45395
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45499
--webserver_port=37755
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:58:41.349215 17013 sys_catalog.cc:263] Verifying existing consensus state
I20260927 06:58:41.353950 17013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: Bootstrap starting.
I20260927 06:58:41.391207 17013 log.cc:824] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: Log is configured to *not* fsync() on all Append() calls
I20260927 06:58:41.412942 17013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 06:58:41.413616 17013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: Bootstrap complete.
I20260927 06:58:41.422834 17013 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef828650a9aa458aa7e0029f55175129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260927 06:58:41.423741 17013 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef828650a9aa458aa7e0029f55175129, State: Initialized, Role: FOLLOWER
I20260927 06:58:41.424414 17013 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef828650a9aa458aa7e0029f55175129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260927 06:58:41.424669 17013 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:58:41.424799 17013 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:58:41.424976 17013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 2 FOLLOWER]: Advancing to term 3
I20260927 06:58:41.428639 17013 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef828650a9aa458aa7e0029f55175129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260927 06:58:41.429101 17013 leader_election.cc:302] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef828650a9aa458aa7e0029f55175129; no voters:
I20260927 06:58:41.429786 17013 leader_election.cc:288] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 06:58:41.429927 17017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 06:58:41.430990 17017 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [term 3 LEADER]: Becoming Leader. State: Replica: ef828650a9aa458aa7e0029f55175129, State: Running, Role: LEADER
I20260927 06:58:41.431577 17017 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef828650a9aa458aa7e0029f55175129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260927 06:58:41.431905 17013 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:58:41.433878 17019 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef828650a9aa458aa7e0029f55175129. Latest consensus state: current_term: 3 leader_uuid: "ef828650a9aa458aa7e0029f55175129" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef828650a9aa458aa7e0029f55175129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } }
I20260927 06:58:41.434602 17019 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [sys.catalog]: This master's current role is: LEADER
I20260927 06:58:41.434175 17018 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ef828650a9aa458aa7e0029f55175129" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef828650a9aa458aa7e0029f55175129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } }
I20260927 06:58:41.435175 17018 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129 [sys.catalog]: This master's current role is: LEADER
I20260927 06:58:41.441494 17023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:58:41.450443 17023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:58:41.452198 17023 catalog_manager.cc:1259] Loaded cluster ID: 4c2bb57af5bf40d98628297bd2f6a74f
I20260927 06:58:41.452334 17023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:58:41.456284 17023 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:58:41.458108 17023 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: Loaded TSK: 0
I20260927 06:58:41.458307 17023 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef828650a9aa458aa7e0029f55175129: Loaded TSK: 1
F20260927 06:58:41.459393 17023 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa8288d29ad google::LogMessage::Fail() at ??:0
@ 0x7fa8288d588a google::LogMessage::SendToLog() at ??:0
@ 0x7fa8288d24de google::LogMessage::Flush() at ??:0
@ 0x7fa8288d3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa82c6367d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa82c62573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa83e3011c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa83e300783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa83e38c2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fa83e38c0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fa83e3d65e9 std::function<>::operator()() at ??:0
@ 0x7fa83e304cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa83e302c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa83e38a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa83e38a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa82b51e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa82b52191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa82b521799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa82b4fceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa82e7d66db start_thread at ??:0
@ 0x7fa82776c61f clone at ??:0
W20260927 06:58:41.784679 17015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:58:41.785012 17015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:58:41.785189 17015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:58:41.792549 17015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:58:41.792692 17015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:58:41.792861 17015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:58:41.802659 17015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37755
--tserver_master_addrs=127.0.97.62:45499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:58:41.804039 17015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:58:41.805364 17015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:58:41.817699 17061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:58:41.817766 17060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:58:41.817976 17063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:58:41.818702 17015 server_base.cc:1034] running on GCE node
I20260927 06:58:41.819247 17015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:58:41.820127 17015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:58:41.821357 17015 hybrid_clock.cc:648] HybridClock initialized: now 1790492321821331 us; error 55 us; skew 500 ppm
I20260927 06:58:41.821674 17015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:58:41 dist-test-slave-f892 krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:58:41.825230 17015 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:58:41.826835 17015 webserver.cc:460] Webserver started at http://127.0.97.1:37755/ using document root <none> and password file <none>
I20260927 06:58:41.827509 17015 fs_manager.cc:359] Metadata directory not provided
I20260927 06:58:41.827652 17015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:58:41.832828 17015 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 06:58:41.835639 17070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:41.836889 17015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 06:58:41.837057 17015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "bc2c9c376ab7450fbd7893fe5112ec48"
format_stamp: "Formatted at 2026-09-27 06:57:55 on dist-test-slave-f892"
server_key: "25459d9c65c7f2354730f04a0b216abf"
server_key_iv: "0af4105c19f416eccdde695320cf6d8e"
server_key_version: "encryptionkey@0"
I20260927 06:58:41.837985 17015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:58:41.856460 17015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:58:41.860127 17015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:58:41.860404 17015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:58:41.861326 17015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:58:41.863276 17077 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260927 06:58:41.869500 17015 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 06:58:41.869666 17015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260927 06:58:41.869818 17015 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 06:58:41.873077 17015 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 06:58:41.873190 17015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.001s
I20260927 06:58:41.873719 17077 tablet_bootstrap.cc:492] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48: Bootstrap starting.
I20260927 06:58:41.901500 17077 log.cc:824] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48: Log is configured to *not* fsync() on all Append() calls
I20260927 06:58:41.915943 17077 tablet_bootstrap.cc:492] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 06:58:41.916728 17077 tablet_bootstrap.cc:492] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48: Bootstrap complete.
I20260927 06:58:41.917322 17015 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45395
I20260927 06:58:41.917413 17184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45395 every 8 connection(s)
I20260927 06:58:41.918282 17077 ts_tablet_manager.cc:1397] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48: Time spent bootstrapping tablet: real 0.045s user 0.036s sys 0.008s
I20260927 06:58:41.918676 17015 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:58:41.923861 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:41.921521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:42657)
0927 06:58:41.922437 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:41.922675 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":358,"threads_started":1}
W20260927 06:58:41.924620 17185 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45499 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45499: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
I20260927 06:58:41.924947 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 17015
I20260927 06:58:41.925594 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:41.925194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:46441)
0927 06:58:41.925375 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:41.925470 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 06:58:41.925673 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38027
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45499
--webserver_port=40523
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:58:41.927016 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:41.926140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:55227)
0927 06:58:41.926722 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:41.926813 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":268,"threads_started":1}
W20260927 06:58:41.927273 17185 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 06:58:41.928145 17077 raft_consensus.cc:348] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } } peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } }
I20260927 06:58:41.929021 17077 raft_consensus.cc:729] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc2c9c376ab7450fbd7893fe5112ec48, State: Initialized, Role: FOLLOWER
I20260927 06:58:41.929677 17077 consensus_queue.cc:260] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } } peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } }
I20260927 06:58:41.931243 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:41.930842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:57913)
0927 06:58:41.931038 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:41.931108 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 06:58:41.931419 17077 ts_tablet_manager.cc:1428] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260927 06:58:41.932277 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:41.931965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:47485)
0927 06:58:41.932101 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:41.932179 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 06:58:42.277287 17190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:58:42.277701 17190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:58:42.277877 17190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:58:42.285071 17190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:58:42.285223 17190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:58:42.285410 17190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:58:42.295277 17190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40523
--tserver_master_addrs=127.0.97.62:45499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:58:42.296620 17190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:58:42.297940 17190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:58:42.310623 17196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:58:42.310669 17198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:58:42.310887 17195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:58:42.311589 17190 server_base.cc:1034] running on GCE node
I20260927 06:58:42.312189 17190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:58:42.313019 17190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:58:42.314213 17190 hybrid_clock.cc:648] HybridClock initialized: now 1790492322314166 us; error 40 us; skew 500 ppm
I20260927 06:58:42.314522 17190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:58:42 dist-test-slave-f892 krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:58:42.317551 17190 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:58:42.318943 17190 webserver.cc:460] Webserver started at http://127.0.97.2:40523/ using document root <none> and password file <none>
I20260927 06:58:42.319526 17190 fs_manager.cc:359] Metadata directory not provided
I20260927 06:58:42.319622 17190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:58:42.324517 17190 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 06:58:42.326960 17205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:42.328235 17190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 06:58:42.328446 17190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "313768f6bc634745b93a16c93271bb36"
format_stamp: "Formatted at 2026-09-27 06:57:56 on dist-test-slave-f892"
server_key: "cfa03080971567a9aa0ad46b37b8dca8"
server_key_iv: "4c6913969f997a7addb7e699620f6a91"
server_key_version: "encryptionkey@0"
I20260927 06:58:42.329437 17190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:58:42.348717 17190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:58:42.352829 17190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:58:42.353103 17190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:58:42.354053 17190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:58:42.355314 17190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:58:42.355441 17190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:42.355582 17190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:58:42.355672 17190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:42.394233 17190 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38027
I20260927 06:58:42.394286 17317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38027 every 8 connection(s)
I20260927 06:58:42.395665 17190 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:58:42.400552 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:42.398690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:36651)
0927 06:58:42.399385 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:42.399575 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":273,"threads_started":1}
I20260927 06:58:42.400717 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 17190
W20260927 06:58:42.401338 17318 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45499 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45499: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
I20260927 06:58:42.401504 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42465
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45499
--webserver_port=38371
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:58:42.402311 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:42.401930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:43367)
0927 06:58:42.402135 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:42.402203 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 06:58:42.403296 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:42.402929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:33453)
0927 06:58:42.403072 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:42.403144 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 06:58:42.403558 17318 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 06:58:42.742542 17321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:58:42.742791 17321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:58:42.742883 17321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:58:42.749915 17321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:58:42.750026 17321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:58:42.750244 17321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:58:42.759752 17321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38371
--tserver_master_addrs=127.0.97.62:45499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:58:42.760993 17321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:58:42.762338 17321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:58:42.778899 17326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:58:42.778976 17327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:58:42.778894 17329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:58:42.779924 17321 server_base.cc:1034] running on GCE node
I20260927 06:58:42.780436 17321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:58:42.781287 17321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:58:42.782474 17321 hybrid_clock.cc:648] HybridClock initialized: now 1790492322782442 us; error 30 us; skew 500 ppm
I20260927 06:58:42.782776 17321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:58:42 dist-test-slave-f892 krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:58:42.785674 17321 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:58:42.787084 17321 webserver.cc:460] Webserver started at http://127.0.97.3:38371/ using document root <none> and password file <none>
I20260927 06:58:42.787734 17321 fs_manager.cc:359] Metadata directory not provided
I20260927 06:58:42.787887 17321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:58:42.793754 17321 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 06:58:42.801608 17336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:42.803846 17321 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 06:58:42.804035 17321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "66daad4cb9374f83a86b311ad2cf6272"
format_stamp: "Formatted at 2026-09-27 06:57:56 on dist-test-slave-f892"
server_key: "93f3078ed97acb1930c71cfbf83b00ab"
server_key_iv: "4f590b0ec4fe35629c42e4b069a71025"
server_key_version: "encryptionkey@0"
I20260927 06:58:42.805157 17321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:58:42.819305 17321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:58:42.823005 17321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:58:42.823283 17321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:58:42.824179 17321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:58:42.825419 17321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:58:42.825543 17321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:42.825663 17321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:58:42.825749 17321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:42.857532 17321 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42465
I20260927 06:58:42.857618 17448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42465 every 8 connection(s)
I20260927 06:58:42.858955 17321 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:58:42.865288 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:42.861740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:53389)
0927 06:58:42.863415 (+ 1675us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:42.863634 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1001,"thread_start_us":756,"threads_started":1}
I20260927 06:58:42.866953 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 17321
W20260927 06:58:42.867405 17449 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45499 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45499: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
I20260927 06:58:42.869511 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:42.869083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:44123)
0927 06:58:42.869310 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:42.869388 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 06:58:42.870676 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:42.870301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:35709)
0927 06:58:42.870456 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:42.870563 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 06:58:42.871297 17449 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 06:58:42.933769 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:42.933080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:41441)
0927 06:58:42.933492 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:42.933588 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":185,"threads_started":1}
I20260927 06:58:43.404940 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:43.404125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:39351)
0927 06:58:43.404559 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:43.404644 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":204,"threads_started":1}
I20260927 06:58:43.782663 17455 raft_consensus.cc:482] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:58:43.783054 17455 raft_consensus.cc:504] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } } peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } }
I20260927 06:58:43.784543 17455 leader_election.cc:288] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 313768f6bc634745b93a16c93271bb36 (127.0.97.2:38027), 66daad4cb9374f83a86b311ad2cf6272 (127.0.97.3:42465)
Sep 27 06:58:43 dist-test-slave-f892 krb5kdc[15865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Sep 27 06:58:43 dist-test-slave-f892 krb5kdc[15865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:58:43.816845 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:43.784891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38027 (local address 127.0.97.1:58257)
0927 06:58:43.797118 (+ 12227us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:43.797160 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 06:58:43.797771 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:58:43.800335 (+ 2564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:58:43.800349 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:58:43.801254 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:58:43.801958 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:43.801980 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:43.803843 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:43.803852 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:43.805078 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:43.805090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:43.806026 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:43.806033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:43.806334 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:43.807156 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:58:43.807173 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:58:43.810071 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:58:43.813453 (+ 3382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:58:43.813458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:58:43.813461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:58:43.813996 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:58:43.814448 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:58:43.814453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:58:43.814456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:58:43.814619 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:58:43.815358 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:58:43.815363 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:58:43.815656 (+ 293us) client_negotiation.cc:769] Sending connection context
0927 06:58:43.815897 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 06:58:43.816371 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12100,"mutex_wait_us":63,"spinlock_wait_cycles":16384,"thread_start_us":12045,"threads_started":1}
I20260927 06:58:43.816869 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:43.785096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58257 (local address 127.0.97.2:38027)
0927 06:58:43.796913 (+ 11817us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:43.796931 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:43.797599 (+ 668us) server_negotiation.cc:407] Connection header received
0927 06:58:43.799556 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:43.799564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:43.799901 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:43.800225 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:58:43.802193 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:43.803618 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:43.805209 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:43.805804 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:43.805963 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:43.810370 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:58:43.810377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:58:43.810387 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:58:43.810422 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:58:43.813291 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:43.814151 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:43.814157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:43.814161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:43.814311 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:43.814756 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:43.814767 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:43.814771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:43.815205 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:58:43.815409 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:43.816119 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 06:58:43.816486 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11608,"thread_start_us":11455,"threads_started":1}
I20260927 06:58:43.816910 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:43.785166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54107 (local address 127.0.97.3:42465)
0927 06:58:43.797551 (+ 12385us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:43.797567 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:43.798200 (+ 633us) server_negotiation.cc:407] Connection header received
0927 06:58:43.799582 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:43.799589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:43.799902 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:43.800214 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:58:43.802226 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:43.803512 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:43.805239 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:43.805835 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:43.806062 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:43.809880 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:58:43.809888 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:58:43.809900 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:58:43.809937 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:58:43.813089 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:43.814117 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:43.814124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:43.814129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:43.814300 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:43.814732 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:43.814739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:43.814747 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:43.815188 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:58:43.815377 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:43.816124 (+ 747us) server_negotiation.cc:299] Negotiation successful
0927 06:58:43.816503 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12228,"thread_start_us":12126,"threads_started":1}
I20260927 06:58:43.816960 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:43.785019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42465 (local address 127.0.97.1:54107)
0927 06:58:43.797983 (+ 12964us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:43.798021 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 06:58:43.798127 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:58:43.800338 (+ 2211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:58:43.800347 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:58:43.801255 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:58:43.801993 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:43.802033 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:43.803686 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:43.803693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:43.805081 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:58:43.805092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:58:43.806104 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:58:43.806115 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:58:43.806349 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:43.807090 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:58:43.807114 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:58:43.809576 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:58:43.813277 (+ 3701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:58:43.813285 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:58:43.813298 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:58:43.813949 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:58:43.814417 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:58:43.814423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:58:43.814430 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:58:43.814613 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:58:43.815326 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:58:43.815334 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:58:43.815631 (+ 297us) client_negotiation.cc:769] Sending connection context
0927 06:58:43.815897 (+ 266us) client_negotiation.cc:240] Negotiation successful
0927 06:58:43.816517 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12843,"mutex_wait_us":353,"spinlock_wait_cycles":37120,"thread_start_us":12674,"threads_started":1}
W20260927 06:58:43.820343 17073 leader_election.cc:341] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 313768f6bc634745b93a16c93271bb36 (127.0.97.2:38027): Not found: Tablet not found: 538ca8ed8e2b430187068ec14ba5a9c0
W20260927 06:58:43.820608 17072 leader_election.cc:341] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66daad4cb9374f83a86b311ad2cf6272 (127.0.97.3:42465): Not found: Tablet not found: 538ca8ed8e2b430187068ec14ba5a9c0
I20260927 06:58:43.820765 17072 leader_election.cc:302] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc2c9c376ab7450fbd7893fe5112ec48; no voters: 313768f6bc634745b93a16c93271bb36, 66daad4cb9374f83a86b311ad2cf6272
I20260927 06:58:43.821175 17455 raft_consensus.cc:2726] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 06:58:43.872540 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:43.871749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:46587)
0927 06:58:43.872172 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:43.872317 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":206,"threads_started":1}
I20260927 06:58:43.935087 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:43.934626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:33139)
0927 06:58:43.934851 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:43.934953 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 06:58:44.406574 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:44.405839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:55765)
0927 06:58:44.406327 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:44.406410 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":274,"threads_started":1}
I20260927 06:58:44.874389 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:44.873457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:37337)
0927 06:58:44.874149 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:44.874233 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":323,"threads_started":1}
I20260927 06:58:44.936682 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:44.935929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:35143)
0927 06:58:44.936420 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:44.936495 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":228,"threads_started":1}
I20260927 06:58:45.326186 17464 raft_consensus.cc:482] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:58:45.326421 17464 raft_consensus.cc:504] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } } peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } }
I20260927 06:58:45.327476 17464 leader_election.cc:288] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 313768f6bc634745b93a16c93271bb36 (127.0.97.2:38027), 66daad4cb9374f83a86b311ad2cf6272 (127.0.97.3:42465)
W20260927 06:58:45.328238 17072 leader_election.cc:341] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66daad4cb9374f83a86b311ad2cf6272 (127.0.97.3:42465): Not found: Tablet not found: 538ca8ed8e2b430187068ec14ba5a9c0
W20260927 06:58:45.328455 17073 leader_election.cc:341] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 313768f6bc634745b93a16c93271bb36 (127.0.97.2:38027): Not found: Tablet not found: 538ca8ed8e2b430187068ec14ba5a9c0
I20260927 06:58:45.328584 17073 leader_election.cc:302] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc2c9c376ab7450fbd7893fe5112ec48; no voters: 313768f6bc634745b93a16c93271bb36, 66daad4cb9374f83a86b311ad2cf6272
I20260927 06:58:45.328819 17464 raft_consensus.cc:2726] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 06:58:45.408133 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:45.407418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:34019)
0927 06:58:45.407853 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:45.407932 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":236,"threads_started":1}
I20260927 06:58:45.876113 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:45.875422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:33107)
0927 06:58:45.875865 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:45.875944 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":232,"threads_started":1}
I20260927 06:58:45.938285 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:45.937579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:60605)
0927 06:58:45.937988 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:45.938128 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":212,"threads_started":1}
I20260927 06:58:46.409905 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:46.409058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:53893)
0927 06:58:46.409616 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:46.409716 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":305,"threads_started":1}
I20260927 06:58:46.878103 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:46.877119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:54267)
0927 06:58:46.877699 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:46.877808 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":280,"threads_started":1}
I20260927 06:58:46.939958 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:46.939276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:44917)
0927 06:58:46.939705 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:46.939781 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":221,"threads_started":1}
I20260927 06:58:47.411442 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:47.410763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:36781)
0927 06:58:47.411215 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:47.411288 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":254,"threads_started":1}
I20260927 06:58:47.879575 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:47.878961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:40189)
0927 06:58:47.879351 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:47.879423 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":208,"threads_started":1}
I20260927 06:58:47.941512 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:47.940792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:50999)
0927 06:58:47.941263 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:47.941351 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":242,"threads_started":1}
I20260927 06:58:48.413036 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:48.412321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:59417)
0927 06:58:48.412760 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:48.412836 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":241,"threads_started":1}
I20260927 06:58:48.634863 17475 raft_consensus.cc:482] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:58:48.635080 17475 raft_consensus.cc:504] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } } peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } }
I20260927 06:58:48.635726 17475 leader_election.cc:288] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 313768f6bc634745b93a16c93271bb36 (127.0.97.2:38027), 66daad4cb9374f83a86b311ad2cf6272 (127.0.97.3:42465)
W20260927 06:58:48.636601 17073 leader_election.cc:341] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 313768f6bc634745b93a16c93271bb36 (127.0.97.2:38027): Not found: Tablet not found: 538ca8ed8e2b430187068ec14ba5a9c0
W20260927 06:58:48.636752 17072 leader_election.cc:341] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66daad4cb9374f83a86b311ad2cf6272 (127.0.97.3:42465): Not found: Tablet not found: 538ca8ed8e2b430187068ec14ba5a9c0
I20260927 06:58:48.636835 17072 leader_election.cc:302] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc2c9c376ab7450fbd7893fe5112ec48; no voters: 313768f6bc634745b93a16c93271bb36, 66daad4cb9374f83a86b311ad2cf6272
I20260927 06:58:48.637030 17475 raft_consensus.cc:2726] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 06:58:48.883430 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:48.882546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:50049)
0927 06:58:48.883063 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:48.883156 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":269,"threads_started":1}
I20260927 06:58:48.943118 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:48.942392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:35919)
0927 06:58:48.942831 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:48.942913 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":221,"threads_started":1}
I20260927 06:58:49.414753 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:49.413922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:34903)
0927 06:58:49.414489 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:49.414572 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":313,"threads_started":1}
I20260927 06:58:49.885190 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:49.884415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:48391)
0927 06:58:49.884928 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:49.885013 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":230,"threads_started":1}
I20260927 06:58:49.944489 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:49.943874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:56763)
0927 06:58:49.944256 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:49.944337 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":203,"threads_started":1}
I20260927 06:58:50.416316 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:50.415642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:42959)
0927 06:58:50.416066 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:50.416151 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":244,"threads_started":1}
I20260927 06:58:50.886825 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:50.886177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:53367)
0927 06:58:50.886562 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:50.886639 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":185,"threads_started":1}
I20260927 06:58:50.946116 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:50.945374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:38117)
0927 06:58:50.945823 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:50.945905 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":235,"threads_started":1}
I20260927 06:58:51.418001 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:51.417271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:58829)
0927 06:58:51.417764 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:51.417850 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":218,"threads_started":1}
I20260927 06:58:51.888406 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:51.887724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:52413)
0927 06:58:51.888183 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:51.888257 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":211,"threads_started":1}
I20260927 06:58:51.947678 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:51.947032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:51943)
0927 06:58:51.947437 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:51.947526 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":198,"threads_started":1}
I20260927 06:58:52.419581 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:52.418948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:56579)
0927 06:58:52.419339 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:52.419414 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":204,"threads_started":1}
I20260927 06:58:52.731431 17488 raft_consensus.cc:482] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:58:52.731649 17488 raft_consensus.cc:504] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } } peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } }
I20260927 06:58:52.732373 17488 leader_election.cc:288] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 313768f6bc634745b93a16c93271bb36 (127.0.97.2:38027), 66daad4cb9374f83a86b311ad2cf6272 (127.0.97.3:42465)
W20260927 06:58:52.733307 17073 leader_election.cc:341] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 313768f6bc634745b93a16c93271bb36 (127.0.97.2:38027): Not found: Tablet not found: 538ca8ed8e2b430187068ec14ba5a9c0
W20260927 06:58:52.733532 17072 leader_election.cc:341] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66daad4cb9374f83a86b311ad2cf6272 (127.0.97.3:42465): Not found: Tablet not found: 538ca8ed8e2b430187068ec14ba5a9c0
I20260927 06:58:52.733708 17072 leader_election.cc:302] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc2c9c376ab7450fbd7893fe5112ec48; no voters: 313768f6bc634745b93a16c93271bb36, 66daad4cb9374f83a86b311ad2cf6272
I20260927 06:58:52.733976 17488 raft_consensus.cc:2726] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 06:58:52.889971 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:52.889335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:39581)
0927 06:58:52.889727 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:52.889800 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":208,"threads_started":1}
I20260927 06:58:52.949301 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:52.948518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:42023)
0927 06:58:52.948991 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:52.949067 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":217,"threads_started":1}
I20260927 06:58:53.421177 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:53.420448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:34163)
0927 06:58:53.420927 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:53.421000 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":245,"threads_started":1}
I20260927 06:58:53.891621 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:53.890976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:54509)
0927 06:58:53.891388 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:53.891461 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":194,"threads_started":1}
I20260927 06:58:53.950832 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:53.950199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:45251)
0927 06:58:53.950596 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:53.950669 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":208,"threads_started":1}
I20260927 06:58:54.423098 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:54.422311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:59171)
0927 06:58:54.422841 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:54.422915 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":310,"threads_started":1}
I20260927 06:58:54.557331 17495 raft_consensus.cc:482] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:58:54.557504 17495 raft_consensus.cc:504] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313768f6bc634745b93a16c93271bb36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38027 } } peers { permanent_uuid: "66daad4cb9374f83a86b311ad2cf6272" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } } peers { permanent_uuid: "bc2c9c376ab7450fbd7893fe5112ec48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45395 } }
I20260927 06:58:54.558202 17495 leader_election.cc:288] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 313768f6bc634745b93a16c93271bb36 (127.0.97.2:38027), 66daad4cb9374f83a86b311ad2cf6272 (127.0.97.3:42465)
W20260927 06:58:54.559106 17073 leader_election.cc:341] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 313768f6bc634745b93a16c93271bb36 (127.0.97.2:38027): Not found: Tablet not found: 538ca8ed8e2b430187068ec14ba5a9c0
W20260927 06:58:54.559259 17072 leader_election.cc:341] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66daad4cb9374f83a86b311ad2cf6272 (127.0.97.3:42465): Not found: Tablet not found: 538ca8ed8e2b430187068ec14ba5a9c0
I20260927 06:58:54.559371 17072 leader_election.cc:302] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc2c9c376ab7450fbd7893fe5112ec48; no voters: 313768f6bc634745b93a16c93271bb36, 66daad4cb9374f83a86b311ad2cf6272
I20260927 06:58:54.559614 17495 raft_consensus.cc:2726] T 538ca8ed8e2b430187068ec14ba5a9c0 P bc2c9c376ab7450fbd7893fe5112ec48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 06:58:54.893203 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:54.892465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:55069)
0927 06:58:54.892964 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:54.893035 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":277,"threads_started":1}
I20260927 06:58:54.952406 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:54.951667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:50401)
0927 06:58:54.952101 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:54.952182 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":231,"threads_started":1}
I20260927 06:58:55.425016 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:55.424041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:52727)
0927 06:58:55.424748 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:55.424852 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":436,"threads_started":1}
I20260927 06:58:55.894815 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:55.894134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:46189)
0927 06:58:55.894563 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:55.894640 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":225,"threads_started":1}
I20260927 06:58:55.953872 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:55.953203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:40093)
0927 06:58:55.953617 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:55.953691 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":203,"threads_started":1}
I20260927 06:58:56.426800 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:56.425904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:55613)
0927 06:58:56.426482 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:56.426586 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":343,"threads_started":1}
I20260927 06:58:56.896457 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:56.895678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:40297)
0927 06:58:56.896130 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:56.896210 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":199,"threads_started":1}
I20260927 06:58:56.955545 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:56.954876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:56627)
0927 06:58:56.955300 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:56.955376 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":205,"threads_started":1}
I20260927 06:58:57.428555 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:57.427860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:38127)
0927 06:58:57.428326 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:57.428408 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":205,"threads_started":1}
I20260927 06:58:57.872785 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 17015
I20260927 06:58:57.887503 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:57.873722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.0.97.1:45395)
0927 06:58:57.874267 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:57.874278 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:57.874314 (+ 36us) server_negotiation.cc:407] Connection header received
0927 06:58:57.874420 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:57.874432 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:57.874723 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:57.874906 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:58:57.876585 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:57.877543 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:57.878890 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:57.879460 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:57.879602 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:57.881997 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:58:57.882045 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:58:57.882064 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:58:57.882112 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:58:57.884646 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:57.885273 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:57.885281 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:57.885285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:57.885431 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:57.885826 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:57.885831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:57.885834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:57.886231 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:58:57.886558 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:57.886893 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 06:58:57.887173 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":254,"threads_started":1}
I20260927 06:58:57.888725 17180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:58:57.898345 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:57.897563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:53457)
0927 06:58:57.898081 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:57.898171 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":268,"threads_started":1}
I20260927 06:58:57.978398 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:57.977315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:52311)
0927 06:58:57.977938 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 06:58:57.978090 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":266,"threads_started":1}
I20260927 06:58:57.978653 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 17015
I20260927 06:58:57.995713 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 17190
I20260927 06:58:58.008832 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:57.996388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43836 (local address 127.0.97.2:38027)
0927 06:58:57.996812 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:57.996817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:57.996831 (+ 14us) server_negotiation.cc:407] Connection header received
0927 06:58:57.996882 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:57.996886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:57.996941 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:57.997055 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:58:57.998345 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:57.999089 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:58.000277 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:58.000704 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:58.000851 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:58.003477 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:58:58.003485 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:58:58.003492 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:58:58.003535 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:58:58.006130 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:58.006767 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:58.006773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:58.006776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:58.006914 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:58.007268 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:58.007274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:58.007277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:58.007574 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:58:58.007867 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:58.008188 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 06:58:58.008419 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":199,"threads_started":1}
I20260927 06:58:58.009563 17313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:58:58.094498 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 17190
I20260927 06:58:58.111825 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 17321
I20260927 06:58:58.125209 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:58.112711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38476 (local address 127.0.97.3:42465)
0927 06:58:58.113212 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:58.113218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:58.113238 (+ 20us) server_negotiation.cc:407] Connection header received
0927 06:58:58.113293 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:58.113297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:58.113362 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:58.113487 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:58:58.114880 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:58.115623 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:58.116833 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:58.117242 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:58.117374 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:58.120100 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:58:58.120105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:58:58.120108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:58:58.120139 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:58:58.122582 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:58.123224 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:58.123232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:58.123238 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:58.123391 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:58.123758 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:58.123763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:58.123768 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:58.124022 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:58:58.124298 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:58.124589 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 06:58:58.124830 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":262,"threads_started":1}
I20260927 06:58:58.125914 17444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:58:58.210572 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 17321
2026-09-27T06:58:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64420 ms)
[----------] 30 tests from SecurityITest (290700 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:58:58 dist-test-slave-f892 krb5kdc[17515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:58:58 dist-test-slave-f892 krb5kdc[17515](info): set up 2 sockets
Sep 27 06:58:58 dist-test-slave-f892 krb5kdc[17515](info): commencing operation
krb5kdc: starting...
W20260927 06:58:59.301986 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:58:59 dist-test-slave-f892 krb5kdc[17515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:58:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:58:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:58:59.535635 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41877
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:58:59.855948 17528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:58:59.856238 17528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:58:59.856422 17528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:58:59.863500 17528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:58:59.863629 17528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:58:59.863730 17528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:58:59.863796 17528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:58:59.863862 17528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:58:59.873308 17528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:58:59.874534 17528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:58:59.875883 17528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:58:59.887563 17533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:58:59.887619 17536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:58:59.887652 17534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:58:59.888144 17528 server_base.cc:1034] running on GCE node
I20260927 06:58:59.888906 17528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:58:59.890195 17528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:58:59.891400 17528 hybrid_clock.cc:648] HybridClock initialized: now 1790492339891364 us; error 37 us; skew 500 ppm
I20260927 06:58:59.891745 17528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:58:59 dist-test-slave-f892 krb5kdc[17515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:58:59.894886 17528 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:58:59.896267 17528 webserver.cc:460] Webserver started at http://127.0.97.62:40477/ using document root <none> and password file <none>
I20260927 06:58:59.896939 17528 fs_manager.cc:359] Metadata directory not provided
I20260927 06:58:59.897064 17528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:58:59.897392 17528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:58:59.900242 17528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "8c08b7d171424841a70cf84b70bdead2"
format_stamp: "Formatted at 2026-09-27 06:58:59 on dist-test-slave-f892"
server_key: "c682369157509b5fc74f23fbed5c5783"
server_key_iv: "b1de14ef632fe8444d75b496d0c7096e"
server_key_version: "encryptionkey@0"
I20260927 06:58:59.901057 17528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "8c08b7d171424841a70cf84b70bdead2"
format_stamp: "Formatted at 2026-09-27 06:58:59 on dist-test-slave-f892"
server_key: "c682369157509b5fc74f23fbed5c5783"
server_key_iv: "b1de14ef632fe8444d75b496d0c7096e"
server_key_version: "encryptionkey@0"
I20260927 06:58:59.906112 17528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 06:58:59.909261 17543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:58:59.910611 17528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:58:59.910787 17528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "8c08b7d171424841a70cf84b70bdead2"
format_stamp: "Formatted at 2026-09-27 06:58:59 on dist-test-slave-f892"
server_key: "c682369157509b5fc74f23fbed5c5783"
server_key_iv: "b1de14ef632fe8444d75b496d0c7096e"
server_key_version: "encryptionkey@0"
I20260927 06:58:59.910964 17528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:58:59.925357 17528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:58:59.929672 17528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:58:59.929971 17528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:58:59.948503 17528 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41877
I20260927 06:58:59.948585 17594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41877 every 8 connection(s)
I20260927 06:58:59.949856 17528 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:58:59.953528 17595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:58:59.956096 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 17528
I20260927 06:58:59.956323 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:58:59.957087 388 external_mini_cluster.cc:1428] Setting key eca81cbb7d7ab175ed6509d1c7767da9
I20260927 06:58:59.965996 17595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2: Bootstrap starting.
I20260927 06:58:59.969705 17595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2: Neither blocks nor log segments found. Creating new log.
I20260927 06:58:59.971050 17595 log.cc:824] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2: Log is configured to *not* fsync() on all Append() calls
Sep 27 06:58:59 dist-test-slave-f892 krb5kdc[17515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:58:59.973942 17595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2: No bootstrap required, opened a new log
I20260927 06:58:59.980182 17598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:58:59.959045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52786 (local address 127.0.97.62:41877)
0927 06:58:59.959872 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0927 06:58:59.959883 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:58:59.959925 (+ 42us) server_negotiation.cc:407] Connection header received
0927 06:58:59.961594 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:58:59.961614 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:58:59.961975 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:58:59.962392 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:58:59.964128 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:59.965350 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:59.967229 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:58:59.967881 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:58:59.968012 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:58:59.972946 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:58:59.972953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:58:59.972966 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:58:59.973000 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:58:59.976139 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:59.976851 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:59.976857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:59.976863 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:59.977029 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:58:59.977449 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:58:59.977453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:58:59.977458 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:58:59.977983 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:58:59.978358 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0927 06:58:59.978806 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 06:58:59.979225 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":408,"threads_started":1}
I20260927 06:58:59.981652 17595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c08b7d171424841a70cf84b70bdead2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41877 } }
I20260927 06:58:59.982036 17595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:58:59.982180 17595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c08b7d171424841a70cf84b70bdead2, State: Initialized, Role: FOLLOWER
I20260927 06:58:59.982795 17595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c08b7d171424841a70cf84b70bdead2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41877 } }
I20260927 06:58:59.983069 17595 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:58:59.983196 17595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:58:59.983315 17595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:58:59.985288 17595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c08b7d171424841a70cf84b70bdead2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41877 } }
I20260927 06:58:59.985692 17595 leader_election.cc:302] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c08b7d171424841a70cf84b70bdead2; no voters:
I20260927 06:58:59.986241 17595 leader_election.cc:288] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:58:59.986380 17600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:58:59.987071 17600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2 [term 1 LEADER]: Becoming Leader. State: Replica: 8c08b7d171424841a70cf84b70bdead2, State: Running, Role: LEADER
I20260927 06:58:59.987639 17600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c08b7d171424841a70cf84b70bdead2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41877 } }
I20260927 06:58:59.988380 17595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:58:59.990723 17602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c08b7d171424841a70cf84b70bdead2. Latest consensus state: current_term: 1 leader_uuid: "8c08b7d171424841a70cf84b70bdead2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c08b7d171424841a70cf84b70bdead2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41877 } } }
I20260927 06:58:59.990660 17601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c08b7d171424841a70cf84b70bdead2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c08b7d171424841a70cf84b70bdead2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41877 } } }
I20260927 06:58:59.991081 17602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2 [sys.catalog]: This master's current role is: LEADER
I20260927 06:58:59.991118 17601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2 [sys.catalog]: This master's current role is: LEADER
I20260927 06:58:59.992784 17609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:58:59.997454 17609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:00.005662 17609 catalog_manager.cc:1347] Generated new cluster ID: 2b98832cd4d0454c82bcef8ed93d8020
I20260927 06:59:00.005784 17609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:00.047338 17609 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:00.048666 17609 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:00.056432 17609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2: Generated new TSK 0
I20260927 06:59:00.057219 17609 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 06:59:00.115214 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41877
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:00.437074 17621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:00.437341 17621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:00.437492 17621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:00.444825 17621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:00.444954 17621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:00.445139 17621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:59:00.455051 17621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:00.456375 17621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:00.457677 17621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:00.470451 17626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:00.470355 17627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:00.470412 17629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:00.471216 17621 server_base.cc:1034] running on GCE node
I20260927 06:59:00.471776 17621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:00.472726 17621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:00.473920 17621 hybrid_clock.cc:648] HybridClock initialized: now 1790492340473886 us; error 33 us; skew 500 ppm
I20260927 06:59:00.474236 17621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:00 dist-test-slave-f892 krb5kdc[17515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:00.477259 17621 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:59:00.478606 17621 webserver.cc:460] Webserver started at http://127.0.97.1:43443/ using document root <none> and password file <none>
I20260927 06:59:00.479272 17621 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:00.479425 17621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:00.479758 17621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:00.482863 17621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "e01cfe60b9c849b68c91dc0fc1aab112"
format_stamp: "Formatted at 2026-09-27 06:59:00 on dist-test-slave-f892"
server_key: "3621062637b2bdbeed5f85fcf12793de"
server_key_iv: "4a52a1031c9ad224ff9e95d795729e8f"
server_key_version: "encryptionkey@0"
I20260927 06:59:00.483541 17621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e01cfe60b9c849b68c91dc0fc1aab112"
format_stamp: "Formatted at 2026-09-27 06:59:00 on dist-test-slave-f892"
server_key: "3621062637b2bdbeed5f85fcf12793de"
server_key_iv: "4a52a1031c9ad224ff9e95d795729e8f"
server_key_version: "encryptionkey@0"
I20260927 06:59:00.488164 17621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 06:59:00.491492 17636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:00.492945 17621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 06:59:00.493124 17621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "e01cfe60b9c849b68c91dc0fc1aab112"
format_stamp: "Formatted at 2026-09-27 06:59:00 on dist-test-slave-f892"
server_key: "3621062637b2bdbeed5f85fcf12793de"
server_key_iv: "4a52a1031c9ad224ff9e95d795729e8f"
server_key_version: "encryptionkey@0"
I20260927 06:59:00.493314 17621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:00.506549 17621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:00.510639 17621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:00.510946 17621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:00.511801 17621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:00.513074 17621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:00.513180 17621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:00.513332 17621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:00.513409 17621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:00.545469 17621 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39093
I20260927 06:59:00.545512 17748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39093 every 8 connection(s)
I20260927 06:59:00.546659 17621 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:59:00.547717 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 17621
I20260927 06:59:00.547935 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:59:00.548831 388 external_mini_cluster.cc:1428] Setting key 1c0b2c0c1d989794c775afd6db0db9f4
Sep 27 06:59:00 dist-test-slave-f892 krb5kdc[17515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:59:00.567689 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:00.549552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49407 (local address 127.0.97.62:41877)
0927 06:59:00.550061 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:00.550067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:00.551127 (+ 1060us) server_negotiation.cc:407] Connection header received
0927 06:59:00.552895 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:00.552903 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:00.553003 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:00.553139 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:00.554594 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:00.555783 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:00.557316 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:00.558110 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:00.558255 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:00.562244 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:00.562249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:00.562252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:00.562281 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:00.564705 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:00.565383 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:00.565394 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:00.565396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:00.565529 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:00.565946 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:00.565950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:00.565957 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:00.566276 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:00.566433 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:00.567088 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 06:59:00.567305 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":249,"threads_started":1}
I20260927 06:59:00.568607 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:00.550132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41877 (local address 127.0.97.1:49407)
0927 06:59:00.550938 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:00.551011 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:00.552638 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:00.553319 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:00.553326 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:00.553839 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:00.554422 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:00.554438 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:00.555951 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:00.555956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:00.557175 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:00.557186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:00.558323 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:00.558333 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:00.558653 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:00.559395 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:00.559429 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:00.561974 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:00.564850 (+ 2876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:00.564858 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:00.564872 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:00.565258 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:00.565640 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:00.565645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:00.565648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:00.565827 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:00.566405 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:00.566413 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:00.566716 (+ 303us) client_negotiation.cc:769] Sending connection context
0927 06:59:00.567002 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 06:59:00.567486 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":354,"threads_started":1}
I20260927 06:59:00.570196 17749 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41877
I20260927 06:59:00.570499 17749 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:00.571195 17749 heartbeater.cc:511] Master 127.0.97.62:41877 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 06:59:00.573298 17560 ts_manager.cc:194] Registered new tserver with Master: e01cfe60b9c849b68c91dc0fc1aab112 (127.0.97.1:39093)
I20260927 06:59:00.574846 17560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 06:59:00.602599 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41877
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:00.924464 17755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:00.924765 17755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:00.924937 17755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:00.931907 17755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:00.932049 17755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:00.932217 17755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:59:00.941433 17755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:00.942644 17755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:00.943904 17755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:00.957614 17761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:00.957844 17763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:00.958076 17760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:00.958386 17755 server_base.cc:1034] running on GCE node
I20260927 06:59:00.959050 17755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:00.960096 17755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:00.961349 17755 hybrid_clock.cc:648] HybridClock initialized: now 1790492340961264 us; error 54 us; skew 500 ppm
I20260927 06:59:00.961678 17755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:00 dist-test-slave-f892 krb5kdc[17515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:00.965286 17755 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:59:00.966779 17755 webserver.cc:460] Webserver started at http://127.0.97.2:45875/ using document root <none> and password file <none>
I20260927 06:59:00.967449 17755 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:00.967592 17755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:00.967893 17755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:00.970783 17755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "2f830f284ccd41d6b55dc80d51485b02"
format_stamp: "Formatted at 2026-09-27 06:59:00 on dist-test-slave-f892"
server_key: "75f095e6ff6f8aa2738097e6ba423ed0"
server_key_iv: "e01d40b9e4c64261d060098a9859f2cd"
server_key_version: "encryptionkey@0"
I20260927 06:59:00.971556 17755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2f830f284ccd41d6b55dc80d51485b02"
format_stamp: "Formatted at 2026-09-27 06:59:00 on dist-test-slave-f892"
server_key: "75f095e6ff6f8aa2738097e6ba423ed0"
server_key_iv: "e01d40b9e4c64261d060098a9859f2cd"
server_key_version: "encryptionkey@0"
I20260927 06:59:00.976451 17755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 06:59:00.979658 17770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:00.980928 17755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 06:59:00.981149 17755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "2f830f284ccd41d6b55dc80d51485b02"
format_stamp: "Formatted at 2026-09-27 06:59:00 on dist-test-slave-f892"
server_key: "75f095e6ff6f8aa2738097e6ba423ed0"
server_key_iv: "e01d40b9e4c64261d060098a9859f2cd"
server_key_version: "encryptionkey@0"
I20260927 06:59:00.981354 17755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:00.999686 17755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:01.003957 17755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:01.004256 17755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:01.005158 17755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:01.006515 17755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:01.006618 17755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:01.006768 17755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:01.006839 17755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:01.039414 17755 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40019
I20260927 06:59:01.039467 17882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40019 every 8 connection(s)
I20260927 06:59:01.040628 17755 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:59:01.044014 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 17755
I20260927 06:59:01.044232 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:59:01.044814 388 external_mini_cluster.cc:1428] Setting key 5fdabfccd545a08859aabdcc906814fa
Sep 27 06:59:01 dist-test-slave-f892 krb5kdc[17515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:59:01.063937 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:01.043028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44137 (local address 127.0.97.62:41877)
0927 06:59:01.043235 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:01.043241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:01.044556 (+ 1315us) server_negotiation.cc:407] Connection header received
0927 06:59:01.046603 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:01.046608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:01.046691 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:01.046795 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:01.048824 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.049593 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.052005 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.052653 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.052772 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:01.057228 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:01.057237 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:01.057245 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:01.057289 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:01.060800 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:01.061482 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:01.061490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:01.061496 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:01.061660 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:01.062074 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:01.062081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:01.062087 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:01.062450 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:01.062595 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:01.063244 (+ 649us) server_negotiation.cc:299] Negotiation successful
0927 06:59:01.063492 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 06:59:01.064534 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:01.043533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41877 (local address 127.0.97.2:44137)
0927 06:59:01.044358 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:01.044404 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:01.046305 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:01.047089 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:01.047101 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:01.047837 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:01.048631 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:01.048658 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.049758 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.049767 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:01.051851 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:01.051868 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.052882 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.052892 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:01.053212 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:01.053970 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:01.054031 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:01.057013 (+ 2982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:01.060933 (+ 3920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:01.060939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:01.060949 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:01.061349 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:01.061775 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:01.061778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:01.061780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:01.061932 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:01.062569 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:01.062575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:01.062870 (+ 295us) client_negotiation.cc:769] Sending connection context
0927 06:59:01.063084 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 06:59:01.063422 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":368,"threads_started":1}
I20260927 06:59:01.066962 17883 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41877
I20260927 06:59:01.067359 17883 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:01.068450 17883 heartbeater.cc:511] Master 127.0.97.62:41877 requested a full tablet report, sending...
I20260927 06:59:01.070510 17560 ts_manager.cc:194] Registered new tserver with Master: 2f830f284ccd41d6b55dc80d51485b02 (127.0.97.2:40019)
I20260927 06:59:01.071920 17560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44137
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 06:59:01.112778 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41877
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:01.431203 17888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:01.431468 17888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:01.431562 17888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:01.438613 17888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:01.438724 17888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:01.438833 17888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:59:01.448333 17888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:01.449493 17888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:01.450820 17888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:01.463203 17893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:01.463157 17896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:01.463239 17894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:01.464375 17888 server_base.cc:1034] running on GCE node
I20260927 06:59:01.464943 17888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:01.465873 17888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:01.467093 17888 hybrid_clock.cc:648] HybridClock initialized: now 1790492341467024 us; error 70 us; skew 500 ppm
I20260927 06:59:01.467388 17888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:01 dist-test-slave-f892 krb5kdc[17515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:01.470427 17888 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:59:01.471810 17888 webserver.cc:460] Webserver started at http://127.0.97.3:42163/ using document root <none> and password file <none>
I20260927 06:59:01.472393 17888 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:01.472487 17888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:01.472736 17888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:01.475485 17888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "e1e6e9efecf94680bb6ab4b57f19ecaa"
format_stamp: "Formatted at 2026-09-27 06:59:01 on dist-test-slave-f892"
server_key: "2719a4d977bc03bcd1da1261ac04d96e"
server_key_iv: "6cef453f8b44ed58b29d88339fa3e7a1"
server_key_version: "encryptionkey@0"
I20260927 06:59:01.476230 17888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e1e6e9efecf94680bb6ab4b57f19ecaa"
format_stamp: "Formatted at 2026-09-27 06:59:01 on dist-test-slave-f892"
server_key: "2719a4d977bc03bcd1da1261ac04d96e"
server_key_iv: "6cef453f8b44ed58b29d88339fa3e7a1"
server_key_version: "encryptionkey@0"
I20260927 06:59:01.481014 17888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260927 06:59:01.484148 17903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:01.485483 17888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 06:59:01.485649 17888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "e1e6e9efecf94680bb6ab4b57f19ecaa"
format_stamp: "Formatted at 2026-09-27 06:59:01 on dist-test-slave-f892"
server_key: "2719a4d977bc03bcd1da1261ac04d96e"
server_key_iv: "6cef453f8b44ed58b29d88339fa3e7a1"
server_key_version: "encryptionkey@0"
I20260927 06:59:01.485863 17888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:01.517534 17888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:01.521174 17888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:01.521452 17888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:01.522375 17888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:01.523602 17888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:01.523682 17888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:01.523861 17888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:01.523912 17888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:01.556797 17888 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34317
I20260927 06:59:01.556844 18015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34317 every 8 connection(s)
I20260927 06:59:01.557978 17888 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:59:01.564893 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 17888
I20260927 06:59:01.565068 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:59:01.565613 388 external_mini_cluster.cc:1428] Setting key 0d338ef35d962996fbf0384b862ef344
Sep 27 06:59:01 dist-test-slave-f892 krb5kdc[17515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:59:01.577304 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:01.560543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37971 (local address 127.0.97.62:41877)
0927 06:59:01.560722 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:01.560726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:01.561781 (+ 1055us) server_negotiation.cc:407] Connection header received
0927 06:59:01.563597 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:01.563604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:01.563687 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:01.563797 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:01.565500 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.566523 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.568293 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.568675 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.568770 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:01.572074 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:01.572079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:01.572083 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:01.572116 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:01.574691 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:01.575392 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:01.575396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:01.575398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:01.575522 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:01.575877 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:01.575880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:01.575882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:01.576153 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:01.576259 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:01.576892 (+ 633us) server_negotiation.cc:299] Negotiation successful
0927 06:59:01.577060 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 06:59:01.577672 17749 heartbeater.cc:503] Master 127.0.97.62:41877 was elected leader, sending a full tablet report...
I20260927 06:59:01.578519 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:01.560968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41877 (local address 127.0.97.3:37971)
0927 06:59:01.561624 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:01.561661 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:01.563330 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:01.564013 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:01.564029 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:01.564492 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:01.565321 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:01.565348 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.566681 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.566686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:01.568132 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:01.568144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.568860 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.568865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:01.569055 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:01.569665 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:01.569686 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:01.571884 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:01.574826 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:01.574833 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:01.574847 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:01.575287 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:01.575613 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:01.575616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:01.575618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:01.575788 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:01.576253 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:01.576258 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:01.576590 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 06:59:01.576812 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 06:59:01.577172 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":288,"threads_started":1}
I20260927 06:59:01.580155 18016 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41877
I20260927 06:59:01.580487 18016 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:01.581408 18016 heartbeater.cc:511] Master 127.0.97.62:41877 requested a full tablet report, sending...
I20260927 06:59:01.582798 17559 ts_manager.cc:194] Registered new tserver with Master: e1e6e9efecf94680bb6ab4b57f19ecaa (127.0.97.3:34317)
I20260927 06:59:01.583606 17559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37971
I20260927 06:59:01.592893 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:59:01.610273 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:01.597290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.0.97.62:41877)
0927 06:59:01.597439 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:01.597443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:01.598052 (+ 609us) server_negotiation.cc:407] Connection header received
0927 06:59:01.598226 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:01.598231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:01.598301 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:01.598397 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:01.599786 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.600490 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.601794 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.602283 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.602413 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:01.604822 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:01.604827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:01.604830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:01.604861 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:01.607615 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:01.608301 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:01.608306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:01.608310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:01.608433 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:01.608796 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:01.608802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:01.608805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:01.609104 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:01.609356 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:01.609697 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 06:59:01.609910 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 06:59:01.613991 17559 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:59:01.620781 17559 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:59:01.643019 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:01.632366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40019 (local address 127.0.0.1:60848)
0927 06:59:01.633460 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:01.633480 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:01.633598 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:01.635759 (+ 2161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:01.635769 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:01.635805 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:01.636168 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:01.636178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.638293 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.638298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:01.640346 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:01.640361 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.641935 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.641941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:01.642169 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:01.642238 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 06:59:01.642366 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 06:59:01.642558 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":2,"thread_start_us":2736,"threads_started":1}
I20260927 06:59:01.643090 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:01.632822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60848 (local address 127.0.97.2:40019)
0927 06:59:01.634293 (+ 1471us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:01.634305 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:01.634339 (+ 34us) server_negotiation.cc:407] Connection header received
0927 06:59:01.634454 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:01.634465 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:01.634755 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:01.636353 (+ 1598us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:01.636570 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.638156 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.640526 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.641737 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.641869 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:01.642079 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:01.642606 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 06:59:01.642794 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1273,"thread_start_us":1115,"threads_started":1}
I20260927 06:59:01.643169 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:01.632799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39093 (local address 127.0.0.1:52454)
0927 06:59:01.635400 (+ 2601us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:01.635421 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:01.635546 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:01.636351 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:01.636355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:01.636381 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:01.636738 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:01.636749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.638520 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.638526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:01.640581 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:01.640595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.642360 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.642366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:01.642528 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:01.642564 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 06:59:01.642666 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 06:59:01.642797 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2456,"thread_start_us":802,"threads_started":1}
I20260927 06:59:01.643987 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:01.631592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34317 (local address 127.0.0.1:58230)
0927 06:59:01.632574 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:01.632624 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:01.632824 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:01.636874 (+ 4050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:01.636879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:01.636906 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:01.637352 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:01.637364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.639400 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.639405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:01.641482 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:01.641496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.643291 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.643297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:01.643463 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:01.643492 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 06:59:01.643560 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 06:59:01.643699 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":462,"threads_started":1}
I20260927 06:59:01.644503 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:01.631906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.0.97.3:34317)
0927 06:59:01.636119 (+ 4213us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:01.636127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:01.636157 (+ 30us) server_negotiation.cc:407] Connection header received
0927 06:59:01.636250 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:01.636259 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:01.636455 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:01.637532 (+ 1077us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:01.637771 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.639237 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.641651 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.643045 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.643798 (+ 753us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:01.643954 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:01.644009 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 06:59:01.644198 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4049,"thread_start_us":3900,"threads_started":1}
I20260927 06:59:01.644743 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:01.633405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.0.97.1:39093)
0927 06:59:01.635796 (+ 2391us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:01.635807 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:01.635837 (+ 30us) server_negotiation.cc:407] Connection header received
0927 06:59:01.635931 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:01.635939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:01.636156 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:01.636348 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:01.636962 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.638401 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.640752 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.642155 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.642278 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:01.642499 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:01.643293 (+ 794us) server_negotiation.cc:299] Negotiation successful
0927 06:59:01.643462 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2219,"thread_start_us":2066,"threads_started":1}
I20260927 06:59:01.647611 17951 tablet_service.cc:1467] Processing CreateTablet for tablet 74e599822fa94328b824c2144a83d903 (DEFAULT_TABLE table=test-table [id=907217a66f5944b5b78e976c15a05561]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:01.647717 17818 tablet_service.cc:1467] Processing CreateTablet for tablet 74e599822fa94328b824c2144a83d903 (DEFAULT_TABLE table=test-table [id=907217a66f5944b5b78e976c15a05561]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:01.648031 17684 tablet_service.cc:1467] Processing CreateTablet for tablet 74e599822fa94328b824c2144a83d903 (DEFAULT_TABLE table=test-table [id=907217a66f5944b5b78e976c15a05561]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:01.648924 17951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74e599822fa94328b824c2144a83d903. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:01.649097 17818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74e599822fa94328b824c2144a83d903. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:01.649276 17684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74e599822fa94328b824c2144a83d903. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:01.660126 18034 tablet_bootstrap.cc:492] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa: Bootstrap starting.
I20260927 06:59:01.663856 18034 tablet_bootstrap.cc:654] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:01.663981 18035 tablet_bootstrap.cc:492] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112: Bootstrap starting.
I20260927 06:59:01.664812 18036 tablet_bootstrap.cc:492] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02: Bootstrap starting.
I20260927 06:59:01.665223 18034 log.cc:824] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:01.667380 18035 tablet_bootstrap.cc:654] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:01.668197 18036 tablet_bootstrap.cc:654] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:01.668536 18034 tablet_bootstrap.cc:492] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa: No bootstrap required, opened a new log
I20260927 06:59:01.668628 18035 log.cc:824] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:01.668813 18034 ts_tablet_manager.cc:1397] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 06:59:01.669349 18036 log.cc:824] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:01.671415 18035 tablet_bootstrap.cc:492] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112: No bootstrap required, opened a new log
I20260927 06:59:01.671759 18035 ts_tablet_manager.cc:1397] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 06:59:01.672261 18036 tablet_bootstrap.cc:492] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02: No bootstrap required, opened a new log
I20260927 06:59:01.672549 18036 ts_tablet_manager.cc:1397] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 06:59:01.676471 18034 raft_consensus.cc:348] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e6e9efecf94680bb6ab4b57f19ecaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34317 } } peers { permanent_uuid: "e01cfe60b9c849b68c91dc0fc1aab112" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39093 } } peers { permanent_uuid: "2f830f284ccd41d6b55dc80d51485b02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 06:59:01.676805 18034 raft_consensus.cc:374] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:01.676918 18034 raft_consensus.cc:729] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1e6e9efecf94680bb6ab4b57f19ecaa, State: Initialized, Role: FOLLOWER
I20260927 06:59:01.677490 18034 consensus_queue.cc:260] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e6e9efecf94680bb6ab4b57f19ecaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34317 } } peers { permanent_uuid: "e01cfe60b9c849b68c91dc0fc1aab112" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39093 } } peers { permanent_uuid: "2f830f284ccd41d6b55dc80d51485b02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 06:59:01.678572 18016 heartbeater.cc:503] Master 127.0.97.62:41877 was elected leader, sending a full tablet report...
I20260927 06:59:01.679257 18034 ts_tablet_manager.cc:1428] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260927 06:59:01.682273 18035 raft_consensus.cc:348] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e6e9efecf94680bb6ab4b57f19ecaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34317 } } peers { permanent_uuid: "e01cfe60b9c849b68c91dc0fc1aab112" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39093 } } peers { permanent_uuid: "2f830f284ccd41d6b55dc80d51485b02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 06:59:01.682756 18035 raft_consensus.cc:374] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:01.682890 18035 raft_consensus.cc:729] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01cfe60b9c849b68c91dc0fc1aab112, State: Initialized, Role: FOLLOWER
I20260927 06:59:01.682746 18036 raft_consensus.cc:348] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e6e9efecf94680bb6ab4b57f19ecaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34317 } } peers { permanent_uuid: "e01cfe60b9c849b68c91dc0fc1aab112" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39093 } } peers { permanent_uuid: "2f830f284ccd41d6b55dc80d51485b02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 06:59:01.683158 18036 raft_consensus.cc:374] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:01.683301 18036 raft_consensus.cc:729] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f830f284ccd41d6b55dc80d51485b02, State: Initialized, Role: FOLLOWER
I20260927 06:59:01.683487 18035 consensus_queue.cc:260] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e6e9efecf94680bb6ab4b57f19ecaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34317 } } peers { permanent_uuid: "e01cfe60b9c849b68c91dc0fc1aab112" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39093 } } peers { permanent_uuid: "2f830f284ccd41d6b55dc80d51485b02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 06:59:01.683844 18036 consensus_queue.cc:260] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e6e9efecf94680bb6ab4b57f19ecaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34317 } } peers { permanent_uuid: "e01cfe60b9c849b68c91dc0fc1aab112" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39093 } } peers { permanent_uuid: "2f830f284ccd41d6b55dc80d51485b02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 06:59:01.685475 18035 ts_tablet_manager.cc:1428] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260927 06:59:01.686690 17883 heartbeater.cc:503] Master 127.0.97.62:41877 was elected leader, sending a full tablet report...
I20260927 06:59:01.687295 18036 ts_tablet_manager.cc:1428] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260927 06:59:01.792690 17884 tablet.cc:2369] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:59:01.799002 17750 tablet.cc:2369] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:59:01.809478 18017 tablet.cc:2369] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:59:01.919024 18042 raft_consensus.cc:482] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:59:01.919360 18042 raft_consensus.cc:504] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e6e9efecf94680bb6ab4b57f19ecaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34317 } } peers { permanent_uuid: "e01cfe60b9c849b68c91dc0fc1aab112" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39093 } } peers { permanent_uuid: "2f830f284ccd41d6b55dc80d51485b02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 06:59:01.921032 18042 leader_election.cc:288] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1e6e9efecf94680bb6ab4b57f19ecaa (127.0.97.3:34317), e01cfe60b9c849b68c91dc0fc1aab112 (127.0.97.1:39093)
I20260927 06:59:01.927855 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:01.921484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44101 (local address 127.0.97.3:34317)
0927 06:59:01.921672 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:01.921676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:01.922069 (+ 393us) server_negotiation.cc:407] Connection header received
0927 06:59:01.922350 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:01.922360 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:01.922447 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:01.922514 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:01.923245 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.924159 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.926161 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.926914 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.927065 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:01.927177 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:01.927460 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 06:59:01.927594 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 06:59:01.928025 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:01.921278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34317 (local address 127.0.97.2:44101)
0927 06:59:01.921904 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:01.921925 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:01.922112 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:01.922702 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:01.922705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:01.922726 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:01.923085 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:01.923094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.924283 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.924291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:01.925917 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:01.925929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.927138 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.927143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:01.927297 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:01.927320 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 06:59:01.927433 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 06:59:01.927555 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":18,"spinlock_wait_cycles":16128,"thread_start_us":345,"threads_started":1}
I20260927 06:59:01.928547 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:01.922402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59063 (local address 127.0.97.1:39093)
0927 06:59:01.922608 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:01.922614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:01.922775 (+ 161us) server_negotiation.cc:407] Connection header received
0927 06:59:01.923012 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:01.923022 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:01.923081 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:01.923182 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:01.923832 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.924839 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.926790 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.927560 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.927709 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:01.927815 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:01.928197 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 06:59:01.928345 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 06:59:01.928707 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:01.922066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39093 (local address 127.0.97.2:59063)
0927 06:59:01.922657 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:01.922680 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:01.922868 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:01.923198 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:01.923202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:01.923225 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:01.923615 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:01.923627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.925006 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.925012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:01.926640 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:01.926652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:01.927787 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:01.927793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:01.928044 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:01.928074 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 06:59:01.928161 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 06:59:01.928281 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":1,"thread_start_us":278,"threads_started":1}
I20260927 06:59:01.929232 17971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e599822fa94328b824c2144a83d903" candidate_uuid: "2f830f284ccd41d6b55dc80d51485b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6e9efecf94680bb6ab4b57f19ecaa" is_pre_election: true
I20260927 06:59:01.929836 17971 raft_consensus.cc:2454] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f830f284ccd41d6b55dc80d51485b02 in term 0.
I20260927 06:59:01.929852 17704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e599822fa94328b824c2144a83d903" candidate_uuid: "2f830f284ccd41d6b55dc80d51485b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01cfe60b9c849b68c91dc0fc1aab112" is_pre_election: true
I20260927 06:59:01.930408 17704 raft_consensus.cc:2454] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f830f284ccd41d6b55dc80d51485b02 in term 0.
I20260927 06:59:01.930636 17772 leader_election.cc:302] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f830f284ccd41d6b55dc80d51485b02, e1e6e9efecf94680bb6ab4b57f19ecaa; no voters:
I20260927 06:59:01.931022 18042 raft_consensus.cc:2781] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:59:01.931145 18042 raft_consensus.cc:482] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:59:01.931293 18042 raft_consensus.cc:3037] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:01.933485 18042 raft_consensus.cc:504] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e6e9efecf94680bb6ab4b57f19ecaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34317 } } peers { permanent_uuid: "e01cfe60b9c849b68c91dc0fc1aab112" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39093 } } peers { permanent_uuid: "2f830f284ccd41d6b55dc80d51485b02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 06:59:01.934165 18042 leader_election.cc:288] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [CANDIDATE]: Term 1 election: Requested vote from peers e1e6e9efecf94680bb6ab4b57f19ecaa (127.0.97.3:34317), e01cfe60b9c849b68c91dc0fc1aab112 (127.0.97.1:39093)
I20260927 06:59:01.934587 17971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e599822fa94328b824c2144a83d903" candidate_uuid: "2f830f284ccd41d6b55dc80d51485b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6e9efecf94680bb6ab4b57f19ecaa"
I20260927 06:59:01.934660 17704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e599822fa94328b824c2144a83d903" candidate_uuid: "2f830f284ccd41d6b55dc80d51485b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01cfe60b9c849b68c91dc0fc1aab112"
I20260927 06:59:01.934803 17971 raft_consensus.cc:3037] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:01.934841 17704 raft_consensus.cc:3037] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:01.938033 17704 raft_consensus.cc:2454] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f830f284ccd41d6b55dc80d51485b02 in term 1.
I20260927 06:59:01.938050 17971 raft_consensus.cc:2454] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f830f284ccd41d6b55dc80d51485b02 in term 1.
I20260927 06:59:01.938460 17772 leader_election.cc:302] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f830f284ccd41d6b55dc80d51485b02, e1e6e9efecf94680bb6ab4b57f19ecaa; no voters:
I20260927 06:59:01.938764 18042 raft_consensus.cc:2781] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:01.939211 18042 raft_consensus.cc:686] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [term 1 LEADER]: Becoming Leader. State: Replica: 2f830f284ccd41d6b55dc80d51485b02, State: Running, Role: LEADER
I20260927 06:59:01.939693 18042 consensus_queue.cc:237] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e6e9efecf94680bb6ab4b57f19ecaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34317 } } peers { permanent_uuid: "e01cfe60b9c849b68c91dc0fc1aab112" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39093 } } peers { permanent_uuid: "2f830f284ccd41d6b55dc80d51485b02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 06:59:01.943945 17560 catalog_manager.cc:5495] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f830f284ccd41d6b55dc80d51485b02 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "2f830f284ccd41d6b55dc80d51485b02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e6e9efecf94680bb6ab4b57f19ecaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01cfe60b9c849b68c91dc0fc1aab112" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f830f284ccd41d6b55dc80d51485b02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } health_report { overall_health: HEALTHY } } }
I20260927 06:59:02.007483 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:02.002072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60850 (local address 127.0.97.2:40019)
0927 06:59:02.002296 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:02.002305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:02.002330 (+ 25us) server_negotiation.cc:407] Connection header received
0927 06:59:02.002598 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:02.002603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:02.002691 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:02.002844 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:02.003202 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:02.003930 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:02.005315 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:02.005727 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:02.005853 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:02.006393 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:02.006894 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:02.007065 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:02.007117 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 06:59:02.007197 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 06:59:02.015915 17704 raft_consensus.cc:1261] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112 [term 1 FOLLOWER]: Refusing update from remote peer 2f830f284ccd41d6b55dc80d51485b02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:02.015915 17971 raft_consensus.cc:1261] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa [term 1 FOLLOWER]: Refusing update from remote peer 2f830f284ccd41d6b55dc80d51485b02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:02.016978 18045 consensus_queue.cc:1035] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1e6e9efecf94680bb6ab4b57f19ecaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:02.017376 18042 consensus_queue.cc:1035] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01cfe60b9c849b68c91dc0fc1aab112" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:02.027872 18048 mvcc.cc:205] Tried to move back new op lower bound from 7333856632885764096 to 7333856632588566528. Current Snapshot: MvccSnapshot[applied={T|T < 7333856632885764096}]
I20260927 06:59:02.028456 18050 mvcc.cc:205] Tried to move back new op lower bound from 7333856632885764096 to 7333856632588566528. Current Snapshot: MvccSnapshot[applied={T|T < 7333856632885764096}]
I20260927 06:59:02.034699 18049 mvcc.cc:205] Tried to move back new op lower bound from 7333856632885764096 to 7333856632588566528. Current Snapshot: MvccSnapshot[applied={T|T < 7333856632885764096}]
I20260927 06:59:02.046705 17559 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:02.047040 17559 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:02.050477 17559 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c08b7d171424841a70cf84b70bdead2: Sending DeleteTablet for 3 replicas of tablet 74e599822fa94328b824c2144a83d903
I20260927 06:59:02.051713 17951 tablet_service.cc:1514] Processing DeleteTablet for tablet 74e599822fa94328b824c2144a83d903 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58230
I20260927 06:59:02.051731 17818 tablet_service.cc:1514] Processing DeleteTablet for tablet 74e599822fa94328b824c2144a83d903 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60848
I20260927 06:59:02.051795 17684 tablet_service.cc:1514] Processing DeleteTablet for tablet 74e599822fa94328b824c2144a83d903 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52454
I20260927 06:59:02.053890 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 17621
I20260927 06:59:02.066283 18059 tablet_replica.cc:330] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112: stopping tablet replica
I20260927 06:59:02.074606 18059 raft_consensus.cc:2229] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:02.075449 18059 raft_consensus.cc:2258] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:02.076524 18058 tablet_replica.cc:330] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa: stopping tablet replica
I20260927 06:59:02.078239 18058 raft_consensus.cc:2229] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:02.078809 18058 raft_consensus.cc:2258] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:02.079470 18060 tablet_replica.cc:330] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02: stopping tablet replica
I20260927 06:59:02.079807 18059 ts_tablet_manager.cc:1905] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:02.079916 18060 raft_consensus.cc:2229] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:59:02.080595 18060 raft_consensus.cc:2258] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:02.082031 18058 ts_tablet_manager.cc:1905] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:02.083679 18060 ts_tablet_manager.cc:1905] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:02.087567 18059 ts_tablet_manager.cc:1918] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:02.087730 18059 log.cc:1196] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/74e599822fa94328b824c2144a83d903
I20260927 06:59:02.088370 18059 ts_tablet_manager.cc:1939] T 74e599822fa94328b824c2144a83d903 P e01cfe60b9c849b68c91dc0fc1aab112: Deleting consensus metadata
I20260927 06:59:02.090682 17545 catalog_manager.cc:4841] TS e01cfe60b9c849b68c91dc0fc1aab112 (127.0.97.1:39093): tablet 74e599822fa94328b824c2144a83d903 (table test-table [id=907217a66f5944b5b78e976c15a05561]) successfully deleted
I20260927 06:59:02.091607 18058 ts_tablet_manager.cc:1918] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:02.091790 18058 log.cc:1196] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/74e599822fa94328b824c2144a83d903
I20260927 06:59:02.092417 18058 ts_tablet_manager.cc:1939] T 74e599822fa94328b824c2144a83d903 P e1e6e9efecf94680bb6ab4b57f19ecaa: Deleting consensus metadata
Sep 27 06:59:02 dist-test-slave-f892 krb5kdc[17515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:59:02.094309 17545 catalog_manager.cc:4841] TS e1e6e9efecf94680bb6ab4b57f19ecaa (127.0.97.3:34317): tablet 74e599822fa94328b824c2144a83d903 (table test-table [id=907217a66f5944b5b78e976c15a05561]) successfully deleted
I20260927 06:59:02.100809 18060 ts_tablet_manager.cc:1918] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:02.101001 18060 log.cc:1196] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/74e599822fa94328b824c2144a83d903
I20260927 06:59:02.101577 18060 ts_tablet_manager.cc:1939] T 74e599822fa94328b824c2144a83d903 P 2f830f284ccd41d6b55dc80d51485b02: Deleting consensus metadata
I20260927 06:59:02.103124 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:02.056601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.0.97.1:39093)
0927 06:59:02.056831 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:02.056840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:02.077609 (+ 20769us) server_negotiation.cc:407] Connection header received
0927 06:59:02.077849 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:02.077856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:02.077940 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:02.078085 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:02.080452 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:02.081586 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:02.083541 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:02.084179 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:02.084278 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:02.095440 (+ 11162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:02.095456 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:02.095474 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:02.095527 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:02.099511 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:02.100393 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:02.100404 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:02.100410 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:02.100588 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:02.101063 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:02.101069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:02.101072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:02.101584 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:02.101896 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:02.102521 (+ 625us) server_negotiation.cc:299] Negotiation successful
0927 06:59:02.102749 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 06:59:02.104450 17744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:02.105028 17547 catalog_manager.cc:4841] TS 2f830f284ccd41d6b55dc80d51485b02 (127.0.97.2:40019): tablet 74e599822fa94328b824c2144a83d903 (table test-table [id=907217a66f5944b5b78e976c15a05561]) successfully deleted
I20260927 06:59:02.231559 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 17621
I20260927 06:59:02.249063 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 17755
Sep 27 06:59:02 dist-test-slave-f892 krb5kdc[17515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:59:02.265758 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:02.250026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60854 (local address 127.0.97.2:40019)
0927 06:59:02.250238 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:02.250244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:02.250265 (+ 21us) server_negotiation.cc:407] Connection header received
0927 06:59:02.250334 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:02.250341 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:02.250432 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:02.250548 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:02.251778 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:02.252971 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:02.254414 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:02.255111 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:02.255228 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:02.259686 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:02.259700 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:02.259716 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:02.259756 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:02.262946 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:02.263748 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:02.263755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:02.263757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:02.263894 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:02.264335 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:02.264340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:02.264342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:02.264690 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:02.264990 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:02.265251 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 06:59:02.265415 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 06:59:02.266517 17878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:02.379180 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 17755
I20260927 06:59:02.398298 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 17888
Sep 27 06:59:02 dist-test-slave-f892 krb5kdc[17515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 06:59:02.412832 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:02.399066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58242 (local address 127.0.97.3:34317)
0927 06:59:02.399270 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:02.399275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:02.399293 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:59:02.399427 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:02.399431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:02.399500 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:02.399586 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:02.400801 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:02.401506 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:02.403001 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:02.403372 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:02.403466 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:02.407714 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:02.407728 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:02.407745 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:02.407779 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:02.410424 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:02.411083 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:02.411090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:02.411092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:02.411215 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:02.411517 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:02.411520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:02.411522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:02.411848 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:02.412081 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:02.412309 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 06:59:02.412467 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 06:59:02.413522 18011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:02.502769 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 17888
I20260927 06:59:02.519472 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 17528
I20260927 06:59:02.520599 17590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:02.630621 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 17528
2026-09-27T06:59:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4422 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T06:59:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:02Z Disabled control of system clock
I20260927 06:59:02.705598 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39509
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 06:59:03.030634 18069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:03.030917 18069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:03.031000 18069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:03.038178 18069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:03.038275 18069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:03.038311 18069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:03.038367 18069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:03.038410 18069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:03.047451 18069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39509
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:03.048732 18069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:03.050451 18069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:03.062233 18074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:03.062136 18075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:03.062192 18077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:03.063024 18069 server_base.cc:1034] running on GCE node
I20260927 06:59:03.063759 18069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:03.065069 18069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:03.066310 18069 hybrid_clock.cc:648] HybridClock initialized: now 1790492343066265 us; error 43 us; skew 500 ppm
I20260927 06:59:03.066659 18069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:59:03.068792 18069 webserver.cc:460] Webserver started at http://127.0.97.62:34467/ using document root <none> and password file <none>
I20260927 06:59:03.069448 18069 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:03.069590 18069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:03.069911 18069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:03.072676 18069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "14d0637e5a424c5297a530130768d569"
format_stamp: "Formatted at 2026-09-27 06:59:03 on dist-test-slave-f892"
server_key: "629367bbcad1ce1809c5c2959f38728b"
server_key_iv: "a0a07f0cc78ae31eb6fd5eb3bd26ccff"
server_key_version: "encryptionkey@0"
I20260927 06:59:03.073444 18069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "14d0637e5a424c5297a530130768d569"
format_stamp: "Formatted at 2026-09-27 06:59:03 on dist-test-slave-f892"
server_key: "629367bbcad1ce1809c5c2959f38728b"
server_key_iv: "a0a07f0cc78ae31eb6fd5eb3bd26ccff"
server_key_version: "encryptionkey@0"
I20260927 06:59:03.078281 18069 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 06:59:03.081473 18083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:03.082774 18069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:59:03.082952 18069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "14d0637e5a424c5297a530130768d569"
format_stamp: "Formatted at 2026-09-27 06:59:03 on dist-test-slave-f892"
server_key: "629367bbcad1ce1809c5c2959f38728b"
server_key_iv: "a0a07f0cc78ae31eb6fd5eb3bd26ccff"
server_key_version: "encryptionkey@0"
I20260927 06:59:03.083158 18069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:03.096709 18069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:03.097507 18069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:03.097810 18069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:03.116062 18069 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39509
I20260927 06:59:03.116087 18134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39509 every 8 connection(s)
I20260927 06:59:03.117390 18069 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:03.120900 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:03.127508 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 18069
I20260927 06:59:03.127697 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:03.128209 388 external_mini_cluster.cc:1428] Setting key 48b94d91e0fbe43223efe8bfb51258a1
I20260927 06:59:03.131664 18135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569: Bootstrap starting.
I20260927 06:59:03.135618 18135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:03.136861 18135 log.cc:824] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:03.139920 18135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569: No bootstrap required, opened a new log
I20260927 06:59:03.144088 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:03.130101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54290 (local address 127.0.97.62:39509)
0927 06:59:03.131070 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:03.131083 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:03.131134 (+ 51us) server_negotiation.cc:407] Connection header received
0927 06:59:03.133583 (+ 2449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:03.133614 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:03.134201 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:03.134619 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:03.135145 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.137129 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.138898 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.139894 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.140087 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:03.141148 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:03.141161 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:03.141179 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:03.141938 (+ 759us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:03.142052 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:03.142168 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:03.142307 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:03.142492 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 06:59:03.142818 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":286,"threads_started":1}
I20260927 06:59:03.148891 18135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d0637e5a424c5297a530130768d569" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39509 } }
I20260927 06:59:03.149295 18135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:03.149443 18135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d0637e5a424c5297a530130768d569, State: Initialized, Role: FOLLOWER
I20260927 06:59:03.150071 18135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d0637e5a424c5297a530130768d569" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39509 } }
I20260927 06:59:03.150338 18135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:03.150488 18135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:03.150661 18135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:03.152791 18135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d0637e5a424c5297a530130768d569" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39509 } }
I20260927 06:59:03.153270 18135 leader_election.cc:302] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14d0637e5a424c5297a530130768d569; no voters:
I20260927 06:59:03.153934 18135 leader_election.cc:288] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:03.154068 18140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:03.154829 18140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569 [term 1 LEADER]: Becoming Leader. State: Replica: 14d0637e5a424c5297a530130768d569, State: Running, Role: LEADER
I20260927 06:59:03.155336 18140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d0637e5a424c5297a530130768d569" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39509 } }
I20260927 06:59:03.155982 18135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:03.157835 18141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14d0637e5a424c5297a530130768d569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d0637e5a424c5297a530130768d569" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39509 } } }
I20260927 06:59:03.158221 18141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:03.163389 18142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14d0637e5a424c5297a530130768d569. Latest consensus state: current_term: 1 leader_uuid: "14d0637e5a424c5297a530130768d569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d0637e5a424c5297a530130768d569" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39509 } } }
I20260927 06:59:03.163672 18142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:03.163900 18146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:03.168830 18146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:03.177052 18146 catalog_manager.cc:1347] Generated new cluster ID: 836a149b26c5470892a17bbc220fb6c5
I20260927 06:59:03.177176 18146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:03.197129 18146 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:03.198411 18146 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:03.204965 18146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569: Generated new TSK 0
I20260927 06:59:03.205929 18146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:03.219436 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39509
--builtin_ntp_servers=127.0.97.20:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 06:59:03.551252 18159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:03.551580 18159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:03.551745 18159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:03.559175 18159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:03.559306 18159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:03.559489 18159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:59:03.568707 18159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:03.569993 18159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:03.571714 18159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:03.584834 18167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:03.584879 18164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:03.584857 18165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:03.585704 18159 server_base.cc:1034] running on GCE node
I20260927 06:59:03.586397 18159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:03.587419 18159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:03.588677 18159 hybrid_clock.cc:648] HybridClock initialized: now 1790492343588599 us; error 74 us; skew 500 ppm
I20260927 06:59:03.588945 18159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:59:03.591188 18159 webserver.cc:460] Webserver started at http://127.0.97.1:44833/ using document root <none> and password file <none>
I20260927 06:59:03.591794 18159 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:03.591878 18159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:03.592137 18159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:03.594918 18159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "1c44168caf1345319552d5e62e748e0b"
format_stamp: "Formatted at 2026-09-27 06:59:03 on dist-test-slave-f892"
server_key: "ced92b9027740521f6a5a21e3dbc9821"
server_key_iv: "5fcab824adfef2d7c412dc670df83058"
server_key_version: "encryptionkey@0"
I20260927 06:59:03.595629 18159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1c44168caf1345319552d5e62e748e0b"
format_stamp: "Formatted at 2026-09-27 06:59:03 on dist-test-slave-f892"
server_key: "ced92b9027740521f6a5a21e3dbc9821"
server_key_iv: "5fcab824adfef2d7c412dc670df83058"
server_key_version: "encryptionkey@0"
I20260927 06:59:03.600279 18159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 06:59:03.603633 18173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:03.604864 18159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 06:59:03.605069 18159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "1c44168caf1345319552d5e62e748e0b"
format_stamp: "Formatted at 2026-09-27 06:59:03 on dist-test-slave-f892"
server_key: "ced92b9027740521f6a5a21e3dbc9821"
server_key_iv: "5fcab824adfef2d7c412dc670df83058"
server_key_version: "encryptionkey@0"
I20260927 06:59:03.605250 18159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:03.620270 18159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:03.620999 18159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:03.621222 18159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:03.622179 18159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:03.623540 18159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:03.623627 18159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:03.623704 18159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:03.623757 18159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:03.657583 18159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42885
I20260927 06:59:03.657692 18285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42885 every 8 connection(s)
I20260927 06:59:03.658919 18159 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:59:03.666311 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 18159
I20260927 06:59:03.666483 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:59:03.667028 388 external_mini_cluster.cc:1428] Setting key e4f301ba0d5e2f0bdc8f88341796b20b
I20260927 06:59:03.669625 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39509
--builtin_ntp_servers=127.0.97.20:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 06:59:03.671708 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:03.661333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45537 (local address 127.0.97.62:39509)
0927 06:59:03.661736 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:03.661741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:03.663055 (+ 1314us) server_negotiation.cc:407] Connection header received
0927 06:59:03.664849 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:03.664853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:03.664920 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:03.665020 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:03.666060 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.667159 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.668727 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.669127 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.669267 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:03.670872 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:03.670878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:03.670883 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:03.670911 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:03.670951 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:03.670963 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:03.671064 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:03.671283 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 06:59:03.671389 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":202,"threads_started":1}
I20260927 06:59:03.672936 18289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:03.661906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39509 (local address 127.0.97.1:45537)
0927 06:59:03.662780 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:03.662819 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:03.664609 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:03.665210 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:03.665218 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:03.665300 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:03.665827 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:03.665842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.667319 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.667323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:03.668568 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:03.668580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.669349 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.669356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:03.669681 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:03.670540 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:59:03.670543 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:03.670581 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:59:03.670655 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:03.671063 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:03.671076 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:03.671087 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 06:59:03.671273 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 06:59:03.671656 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":248,"threads_started":1}
I20260927 06:59:03.674700 18286 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39509
I20260927 06:59:03.674988 18286 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:03.675788 18286 heartbeater.cc:511] Master 127.0.97.62:39509 requested a full tablet report, sending...
I20260927 06:59:03.677902 18100 ts_manager.cc:194] Registered new tserver with Master: 1c44168caf1345319552d5e62e748e0b (127.0.97.1:42885)
I20260927 06:59:03.679411 18100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:45537
W20260927 06:59:03.989048 18290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:03.989337 18290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:03.989501 18290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:03.996958 18290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:03.997083 18290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:03.997262 18290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:59:04.006235 18290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:04.007498 18290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:04.009217 18290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:04.022495 18295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:04.022593 18296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:04.022614 18298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:04.023473 18290 server_base.cc:1034] running on GCE node
I20260927 06:59:04.024060 18290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:04.024976 18290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:04.026171 18290 hybrid_clock.cc:648] HybridClock initialized: now 1790492344026134 us; error 34 us; skew 500 ppm
I20260927 06:59:04.026490 18290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:59:04.028609 18290 webserver.cc:460] Webserver started at http://127.0.97.2:42913/ using document root <none> and password file <none>
I20260927 06:59:04.029287 18290 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:04.029381 18290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:04.029747 18290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:04.032588 18290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "da956d9bcba5409390340ecb28ac08f9"
format_stamp: "Formatted at 2026-09-27 06:59:04 on dist-test-slave-f892"
server_key: "dd5b21f4c360bb6a86d9bfb5e977cdfc"
server_key_iv: "a2bf14d3b96ba03b69220e6e9773eab7"
server_key_version: "encryptionkey@0"
I20260927 06:59:04.033346 18290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "da956d9bcba5409390340ecb28ac08f9"
format_stamp: "Formatted at 2026-09-27 06:59:04 on dist-test-slave-f892"
server_key: "dd5b21f4c360bb6a86d9bfb5e977cdfc"
server_key_iv: "a2bf14d3b96ba03b69220e6e9773eab7"
server_key_version: "encryptionkey@0"
I20260927 06:59:04.038249 18290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 06:59:04.041532 18304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:04.042891 18290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:59:04.043051 18290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "da956d9bcba5409390340ecb28ac08f9"
format_stamp: "Formatted at 2026-09-27 06:59:04 on dist-test-slave-f892"
server_key: "dd5b21f4c360bb6a86d9bfb5e977cdfc"
server_key_iv: "a2bf14d3b96ba03b69220e6e9773eab7"
server_key_version: "encryptionkey@0"
I20260927 06:59:04.043272 18290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:04.061028 18290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:04.061731 18290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:04.061968 18290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:04.062979 18290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:04.064455 18290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:04.064543 18290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:04.064630 18290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:04.064687 18290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:04.096868 18290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32771
I20260927 06:59:04.096923 18416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32771 every 8 connection(s)
I20260927 06:59:04.098179 18290 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:59:04.103519 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 18290
I20260927 06:59:04.103698 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:59:04.104322 388 external_mini_cluster.cc:1428] Setting key f7710bdee94a9140acf3959fc35de7d6
I20260927 06:59:04.107403 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39509
--builtin_ntp_servers=127.0.97.20:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 06:59:04.111968 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:04.100620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46821 (local address 127.0.97.62:39509)
0927 06:59:04.100788 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:04.100792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:04.102079 (+ 1287us) server_negotiation.cc:407] Connection header received
0927 06:59:04.104334 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:04.104338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:04.104452 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:04.104542 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:04.105728 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.106558 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.109027 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.109523 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.109621 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:04.111256 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:04.111264 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:04.111267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:04.111285 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:04.111317 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:04.111328 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:04.111399 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:04.111648 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 06:59:04.111726 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 06:59:04.113435 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:04.101057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39509 (local address 127.0.97.2:46821)
0927 06:59:04.101906 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:04.101939 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:04.104027 (+ 2088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:04.104807 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:04.104818 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:04.104948 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:04.105563 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:04.105586 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.106712 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.106719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:04.108878 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:04.108905 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.109788 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.109797 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:04.110148 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:04.110900 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:59:04.110906 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:04.110956 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:59:04.111056 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:04.111426 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:04.111439 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:04.111450 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 06:59:04.111630 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 06:59:04.112064 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":260,"threads_started":1}
I20260927 06:59:04.114838 18417 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39509
I20260927 06:59:04.115134 18417 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:04.116083 18417 heartbeater.cc:511] Master 127.0.97.62:39509 requested a full tablet report, sending...
I20260927 06:59:04.117550 18100 ts_manager.cc:194] Registered new tserver with Master: da956d9bcba5409390340ecb28ac08f9 (127.0.97.2:32771)
I20260927 06:59:04.118448 18100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:46821
W20260927 06:59:04.437465 18420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:04.437767 18420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:04.437973 18420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:04.445142 18420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:04.445268 18420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:04.445487 18420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:59:04.454614 18420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:04.455897 18420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:04.457774 18420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:04.470628 18425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:04.470567 18428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:04.470672 18426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:04.471469 18420 server_base.cc:1034] running on GCE node
I20260927 06:59:04.472056 18420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:04.472945 18420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:04.474097 18420 hybrid_clock.cc:648] HybridClock initialized: now 1790492344474068 us; error 29 us; skew 500 ppm
I20260927 06:59:04.474396 18420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:59:04.476500 18420 webserver.cc:460] Webserver started at http://127.0.97.3:39375/ using document root <none> and password file <none>
I20260927 06:59:04.477145 18420 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:04.477252 18420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:04.477561 18420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:04.480266 18420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "214ff3d4d4cc46b982f1e4b28c6bca6a"
format_stamp: "Formatted at 2026-09-27 06:59:04 on dist-test-slave-f892"
server_key: "761f8d273f1744c3d1461117e0fe2070"
server_key_iv: "b38293c1e418d92c20230b4a91283bc5"
server_key_version: "encryptionkey@0"
I20260927 06:59:04.481014 18420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "214ff3d4d4cc46b982f1e4b28c6bca6a"
format_stamp: "Formatted at 2026-09-27 06:59:04 on dist-test-slave-f892"
server_key: "761f8d273f1744c3d1461117e0fe2070"
server_key_iv: "b38293c1e418d92c20230b4a91283bc5"
server_key_version: "encryptionkey@0"
I20260927 06:59:04.485535 18420 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 06:59:04.488739 18434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:04.489954 18420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 06:59:04.490165 18420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "214ff3d4d4cc46b982f1e4b28c6bca6a"
format_stamp: "Formatted at 2026-09-27 06:59:04 on dist-test-slave-f892"
server_key: "761f8d273f1744c3d1461117e0fe2070"
server_key_iv: "b38293c1e418d92c20230b4a91283bc5"
server_key_version: "encryptionkey@0"
I20260927 06:59:04.490389 18420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:04.504120 18420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:04.504810 18420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:04.505070 18420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:04.505930 18420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:04.507283 18420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:04.507397 18420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:04.507519 18420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:04.507609 18420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:04.540592 18420 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46823
I20260927 06:59:04.540638 18546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46823 every 8 connection(s)
I20260927 06:59:04.541860 18420 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:59:04.548022 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 18420
I20260927 06:59:04.548203 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:59:04.548664 388 external_mini_cluster.cc:1428] Setting key 5c35a70d153d6ee9fb6c3b3dcad40a5a
I20260927 06:59:04.554271 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:04.544403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52243 (local address 127.0.97.62:39509)
0927 06:59:04.544598 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:04.544602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:04.545823 (+ 1221us) server_negotiation.cc:407] Connection header received
0927 06:59:04.547731 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:04.547739 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:04.547853 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:04.547988 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:04.549083 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.550123 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.551895 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.552275 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.552370 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:04.553589 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:04.553593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:04.553596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:04.553612 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:04.553637 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:04.553648 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:04.553712 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:04.553899 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 06:59:04.553965 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 06:59:04.555148 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:04.544867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39509 (local address 127.0.97.3:52243)
0927 06:59:04.545682 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:04.545716 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:04.547467 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:04.548209 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:04.548217 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:04.548331 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:04.548910 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:04.548927 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.550282 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.550292 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:04.551766 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:04.551778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.552449 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.552463 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:04.552642 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:04.553330 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:59:04.553332 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:04.553367 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:59:04.553434 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:04.553729 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:04.553735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:04.553740 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 06:59:04.553869 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 06:59:04.554188 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":249,"threads_started":1}
I20260927 06:59:04.556347 18547 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39509
I20260927 06:59:04.556618 18547 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:04.557371 18547 heartbeater.cc:511] Master 127.0.97.62:39509 requested a full tablet report, sending...
I20260927 06:59:04.558648 18100 ts_manager.cc:194] Registered new tserver with Master: 214ff3d4d4cc46b982f1e4b28c6bca6a (127.0.97.3:46823)
I20260927 06:59:04.559438 18100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:52243
I20260927 06:59:04.563176 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:59:04.572852 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:04.567021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.0.97.62:39509)
0927 06:59:04.567186 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:04.567190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:04.567595 (+ 405us) server_negotiation.cc:407] Connection header received
0927 06:59:04.567771 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:04.567790 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:04.567890 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:04.567999 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:04.568442 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.569107 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.570573 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.571026 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.571152 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:04.572120 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:04.572126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:04.572130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:04.572156 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:04.572184 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:04.572196 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:04.572317 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:04.572470 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 06:59:04.572552 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 06:59:04.576484 18100 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:59:04.583074 18100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:59:04.617882 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:04.594511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32771 (local address 127.0.0.1:43498)
0927 06:59:04.601669 (+ 7158us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:04.601726 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:04.601923 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:04.610516 (+ 8593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:04.610520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:04.610538 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:04.611058 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:04.611414 (+ 356us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.612754 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.612761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:04.615310 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:04.615326 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.616810 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.616815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:04.617009 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:04.617056 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 06:59:04.617223 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 06:59:04.617431 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6844,"mutex_wait_us":2,"thread_start_us":6694,"threads_started":1}
I20260927 06:59:04.618177 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:04.594794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43498 (local address 127.0.97.2:32771)
0927 06:59:04.609876 (+ 15082us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:04.609884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:04.609915 (+ 31us) server_negotiation.cc:407] Connection header received
0927 06:59:04.610015 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:04.610039 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:04.610289 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:04.610595 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:04.611604 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.612614 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.615545 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.616615 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.616784 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:04.617061 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:04.617265 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 06:59:04.617802 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14917,"thread_start_us":14788,"threads_started":1}
I20260927 06:59:04.618772 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:04.594748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46823 (local address 127.0.0.1:59478)
0927 06:59:04.607325 (+ 12577us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:04.607354 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:04.607502 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:04.610487 (+ 2985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:04.610494 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:04.610520 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:04.611058 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:04.611075 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.613186 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.613196 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:04.615554 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:04.615567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.617742 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.617748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:04.618052 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:04.618205 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 06:59:04.618282 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 06:59:04.618432 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12399,"mutex_wait_us":209,"thread_start_us":12854,"threads_started":1}
I20260927 06:59:04.618935 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:04.602446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.0.97.1:42885)
0927 06:59:04.609493 (+ 7047us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:04.609504 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:04.609533 (+ 29us) server_negotiation.cc:407] Connection header received
0927 06:59:04.609637 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:04.609645 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:04.609846 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:04.610139 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:04.611675 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.613240 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.615844 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.617239 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.617354 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:04.617572 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:04.618415 (+ 843us) server_negotiation.cc:299] Negotiation successful
0927 06:59:04.618592 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6878,"thread_start_us":6771,"threads_started":1}
I20260927 06:59:04.619494 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:04.607364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42885 (local address 127.0.0.1:45216)
0927 06:59:04.608912 (+ 1548us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:04.608934 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:04.609040 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:04.610466 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:04.610474 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:04.610504 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:04.611373 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:04.611389 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.613383 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.613388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:04.615644 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:04.615659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.617742 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.617748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:04.618052 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:04.618207 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 06:59:04.618282 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 06:59:04.619164 (+ 882us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1434,"mutex_wait_us":837,"thread_start_us":1312,"threads_started":1}
I20260927 06:59:04.619729 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:04.595204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59478 (local address 127.0.97.3:46823)
0927 06:59:04.608912 (+ 13708us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:04.608931 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:04.608980 (+ 49us) server_negotiation.cc:407] Connection header received
0927 06:59:04.609077 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:04.609090 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:04.609418 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:04.609693 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:04.611276 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.613045 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.615738 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.617195 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.617339 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:04.617614 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:04.619181 (+ 1567us) server_negotiation.cc:299] Negotiation successful
0927 06:59:04.619344 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13544,"thread_start_us":13250,"threads_started":1}
I20260927 06:59:04.622437 18352 tablet_service.cc:1467] Processing CreateTablet for tablet 7780bbf01ea24b54b161f153383a375b (DEFAULT_TABLE table=test-table [id=b5fecfea767945a5b8a2145700864032]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:04.623163 18221 tablet_service.cc:1467] Processing CreateTablet for tablet 7780bbf01ea24b54b161f153383a375b (DEFAULT_TABLE table=test-table [id=b5fecfea767945a5b8a2145700864032]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:04.623163 18482 tablet_service.cc:1467] Processing CreateTablet for tablet 7780bbf01ea24b54b161f153383a375b (DEFAULT_TABLE table=test-table [id=b5fecfea767945a5b8a2145700864032]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:04.623728 18352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7780bbf01ea24b54b161f153383a375b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:04.624580 18482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7780bbf01ea24b54b161f153383a375b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:04.624593 18221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7780bbf01ea24b54b161f153383a375b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:04.634270 18565 tablet_bootstrap.cc:492] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9: Bootstrap starting.
I20260927 06:59:04.637380 18565 tablet_bootstrap.cc:654] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:04.638545 18565 log.cc:824] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:04.638717 18566 tablet_bootstrap.cc:492] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a: Bootstrap starting.
I20260927 06:59:04.642165 18567 tablet_bootstrap.cc:492] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b: Bootstrap starting.
I20260927 06:59:04.642232 18565 tablet_bootstrap.cc:492] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9: No bootstrap required, opened a new log
I20260927 06:59:04.642315 18566 tablet_bootstrap.cc:654] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:04.642508 18565 ts_tablet_manager.cc:1397] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260927 06:59:04.643443 18566 log.cc:824] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:04.644832 18567 tablet_bootstrap.cc:654] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:04.645916 18567 log.cc:824] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:04.646697 18566 tablet_bootstrap.cc:492] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a: No bootstrap required, opened a new log
I20260927 06:59:04.646970 18566 ts_tablet_manager.cc:1397] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260927 06:59:04.648451 18567 tablet_bootstrap.cc:492] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b: No bootstrap required, opened a new log
I20260927 06:59:04.648625 18567 ts_tablet_manager.cc:1397] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 06:59:04.652658 18565 raft_consensus.cc:348] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da956d9bcba5409390340ecb28ac08f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "214ff3d4d4cc46b982f1e4b28c6bca6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46823 } } peers { permanent_uuid: "1c44168caf1345319552d5e62e748e0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42885 } }
I20260927 06:59:04.653095 18565 raft_consensus.cc:374] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:04.653227 18565 raft_consensus.cc:729] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da956d9bcba5409390340ecb28ac08f9, State: Initialized, Role: FOLLOWER
I20260927 06:59:04.653841 18565 consensus_queue.cc:260] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da956d9bcba5409390340ecb28ac08f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "214ff3d4d4cc46b982f1e4b28c6bca6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46823 } } peers { permanent_uuid: "1c44168caf1345319552d5e62e748e0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42885 } }
I20260927 06:59:04.655107 18417 heartbeater.cc:503] Master 127.0.97.62:39509 was elected leader, sending a full tablet report...
I20260927 06:59:04.655773 18565 ts_tablet_manager.cc:1428] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 06:59:04.656587 18567 raft_consensus.cc:348] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da956d9bcba5409390340ecb28ac08f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "214ff3d4d4cc46b982f1e4b28c6bca6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46823 } } peers { permanent_uuid: "1c44168caf1345319552d5e62e748e0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42885 } }
I20260927 06:59:04.658071 18566 raft_consensus.cc:348] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da956d9bcba5409390340ecb28ac08f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "214ff3d4d4cc46b982f1e4b28c6bca6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46823 } } peers { permanent_uuid: "1c44168caf1345319552d5e62e748e0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42885 } }
I20260927 06:59:04.658551 18566 raft_consensus.cc:374] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:04.658679 18567 raft_consensus.cc:374] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:04.658707 18566 raft_consensus.cc:729] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214ff3d4d4cc46b982f1e4b28c6bca6a, State: Initialized, Role: FOLLOWER
I20260927 06:59:04.658888 18567 raft_consensus.cc:729] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c44168caf1345319552d5e62e748e0b, State: Initialized, Role: FOLLOWER
I20260927 06:59:04.659405 18566 consensus_queue.cc:260] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da956d9bcba5409390340ecb28ac08f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "214ff3d4d4cc46b982f1e4b28c6bca6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46823 } } peers { permanent_uuid: "1c44168caf1345319552d5e62e748e0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42885 } }
I20260927 06:59:04.659606 18567 consensus_queue.cc:260] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da956d9bcba5409390340ecb28ac08f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "214ff3d4d4cc46b982f1e4b28c6bca6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46823 } } peers { permanent_uuid: "1c44168caf1345319552d5e62e748e0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42885 } }
I20260927 06:59:04.666688 18547 heartbeater.cc:503] Master 127.0.97.62:39509 was elected leader, sending a full tablet report...
I20260927 06:59:04.667763 18566 ts_tablet_manager.cc:1428] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a: Time spent starting tablet: real 0.021s user 0.013s sys 0.006s
I20260927 06:59:04.668953 18286 heartbeater.cc:503] Master 127.0.97.62:39509 was elected leader, sending a full tablet report...
I20260927 06:59:04.669603 18567 ts_tablet_manager.cc:1428] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b: Time spent starting tablet: real 0.021s user 0.011s sys 0.006s
I20260927 06:59:04.689425 18573 raft_consensus.cc:482] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:59:04.689697 18573 raft_consensus.cc:504] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da956d9bcba5409390340ecb28ac08f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "214ff3d4d4cc46b982f1e4b28c6bca6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46823 } } peers { permanent_uuid: "1c44168caf1345319552d5e62e748e0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42885 } }
I20260927 06:59:04.691241 18573 leader_election.cc:288] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da956d9bcba5409390340ecb28ac08f9 (127.0.97.2:32771), 214ff3d4d4cc46b982f1e4b28c6bca6a (127.0.97.3:46823)
I20260927 06:59:04.700173 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:04.691617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48991 (local address 127.0.97.2:32771)
0927 06:59:04.691809 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:04.691815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:04.692593 (+ 778us) server_negotiation.cc:407] Connection header received
0927 06:59:04.692762 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:04.692767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:04.692856 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:04.692946 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:04.693766 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.694887 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.697823 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.699073 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.699203 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:04.699295 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:04.699782 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 06:59:04.699910 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 06:59:04.700170 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:04.691617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40987 (local address 127.0.97.3:46823)
0927 06:59:04.691808 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:04.691815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:04.692577 (+ 762us) server_negotiation.cc:407] Connection header received
0927 06:59:04.692765 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:04.692770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:04.692845 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:04.692961 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:04.693775 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.694895 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.697825 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.699075 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.699203 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:04.699308 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:04.699782 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 06:59:04.699910 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 06:59:04.700385 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:04.691502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46823 (local address 127.0.97.1:40987)
0927 06:59:04.692432 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:04.692457 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:04.692641 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:04.692975 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:04.692981 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:04.693014 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:04.693617 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:04.693628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.695025 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.695031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:04.697606 (+ 2575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:04.697621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.699305 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.699311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:04.699581 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:04.699650 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 06:59:04.699740 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 06:59:04.699922 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":309,"spinlock_wait_cycles":32384,"thread_start_us":482,"threads_started":1}
I20260927 06:59:04.700410 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:04.691569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32771 (local address 127.0.97.1:48991)
0927 06:59:04.692484 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:04.692499 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:04.692606 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:04.692975 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:04.692981 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:04.693020 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:04.693589 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:04.693607 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.695025 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.695031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:04.697607 (+ 2576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:04.697621 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.699286 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.699294 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:04.699587 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:04.699646 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 06:59:04.699740 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 06:59:04.699922 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":342,"spinlock_wait_cycles":2944,"thread_start_us":438,"threads_started":1}
I20260927 06:59:04.701514 18372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7780bbf01ea24b54b161f153383a375b" candidate_uuid: "1c44168caf1345319552d5e62e748e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da956d9bcba5409390340ecb28ac08f9" is_pre_election: true
I20260927 06:59:04.701507 18502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7780bbf01ea24b54b161f153383a375b" candidate_uuid: "1c44168caf1345319552d5e62e748e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214ff3d4d4cc46b982f1e4b28c6bca6a" is_pre_election: true
I20260927 06:59:04.701962 18502 raft_consensus.cc:2454] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c44168caf1345319552d5e62e748e0b in term 0.
I20260927 06:59:04.701977 18372 raft_consensus.cc:2454] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c44168caf1345319552d5e62e748e0b in term 0.
I20260927 06:59:04.702692 18174 leader_election.cc:302] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c44168caf1345319552d5e62e748e0b, 214ff3d4d4cc46b982f1e4b28c6bca6a, da956d9bcba5409390340ecb28ac08f9; no voters:
I20260927 06:59:04.703055 18573 raft_consensus.cc:2781] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:59:04.703222 18573 raft_consensus.cc:482] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:59:04.703325 18573 raft_consensus.cc:3037] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:04.705650 18573 raft_consensus.cc:504] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da956d9bcba5409390340ecb28ac08f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "214ff3d4d4cc46b982f1e4b28c6bca6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46823 } } peers { permanent_uuid: "1c44168caf1345319552d5e62e748e0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42885 } }
I20260927 06:59:04.706349 18573 leader_election.cc:288] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [CANDIDATE]: Term 1 election: Requested vote from peers da956d9bcba5409390340ecb28ac08f9 (127.0.97.2:32771), 214ff3d4d4cc46b982f1e4b28c6bca6a (127.0.97.3:46823)
I20260927 06:59:04.706672 18372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7780bbf01ea24b54b161f153383a375b" candidate_uuid: "1c44168caf1345319552d5e62e748e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da956d9bcba5409390340ecb28ac08f9"
I20260927 06:59:04.706748 18502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7780bbf01ea24b54b161f153383a375b" candidate_uuid: "1c44168caf1345319552d5e62e748e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214ff3d4d4cc46b982f1e4b28c6bca6a"
I20260927 06:59:04.706857 18372 raft_consensus.cc:3037] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:04.706936 18502 raft_consensus.cc:3037] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:04.709172 18502 raft_consensus.cc:2454] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c44168caf1345319552d5e62e748e0b in term 1.
I20260927 06:59:04.709179 18372 raft_consensus.cc:2454] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c44168caf1345319552d5e62e748e0b in term 1.
I20260927 06:59:04.709578 18176 leader_election.cc:302] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c44168caf1345319552d5e62e748e0b, da956d9bcba5409390340ecb28ac08f9; no voters:
I20260927 06:59:04.709836 18573 raft_consensus.cc:2781] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:04.710297 18573 raft_consensus.cc:686] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [term 1 LEADER]: Becoming Leader. State: Replica: 1c44168caf1345319552d5e62e748e0b, State: Running, Role: LEADER
I20260927 06:59:04.710769 18573 consensus_queue.cc:237] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da956d9bcba5409390340ecb28ac08f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "214ff3d4d4cc46b982f1e4b28c6bca6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46823 } } peers { permanent_uuid: "1c44168caf1345319552d5e62e748e0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42885 } }
I20260927 06:59:04.714942 18099 catalog_manager.cc:5495] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c44168caf1345319552d5e62e748e0b (127.0.97.1). New cstate: current_term: 1 leader_uuid: "1c44168caf1345319552d5e62e748e0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da956d9bcba5409390340ecb28ac08f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "214ff3d4d4cc46b982f1e4b28c6bca6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c44168caf1345319552d5e62e748e0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42885 } health_report { overall_health: HEALTHY } } }
I20260927 06:59:04.740662 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:04.735769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45226 (local address 127.0.97.1:42885)
0927 06:59:04.735958 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:04.735962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:04.735980 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:59:04.736062 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:04.736065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:04.736153 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:04.736272 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:04.736582 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.737303 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.738727 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.739132 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.739227 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:04.739654 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:04.740142 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:04.740246 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:04.740383 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 06:59:04.740464 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 06:59:04.748466 18372 raft_consensus.cc:1261] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c44168caf1345319552d5e62e748e0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:04.748517 18502 raft_consensus.cc:1261] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a [term 1 FOLLOWER]: Refusing update from remote peer 1c44168caf1345319552d5e62e748e0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:04.749392 18576 consensus_queue.cc:1035] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "da956d9bcba5409390340ecb28ac08f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:04.749729 18573 consensus_queue.cc:1035] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "214ff3d4d4cc46b982f1e4b28c6bca6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:04.760463 18579 mvcc.cc:205] Tried to move back new op lower bound from 7333856644080160768 to 7333856643938934784. Current Snapshot: MvccSnapshot[applied={T|T < 7333856644080160768}]
I20260927 06:59:04.760535 18581 mvcc.cc:205] Tried to move back new op lower bound from 7333856644080160768 to 7333856643938934784. Current Snapshot: MvccSnapshot[applied={T|T < 7333856644080160768}]
I20260927 06:59:04.774943 18100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:04.775189 18100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:04.778570 18100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14d0637e5a424c5297a530130768d569: Sending DeleteTablet for 3 replicas of tablet 7780bbf01ea24b54b161f153383a375b
I20260927 06:59:04.779646 18352 tablet_service.cc:1514] Processing DeleteTablet for tablet 7780bbf01ea24b54b161f153383a375b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:04 UTC) from {username='slave'} at 127.0.0.1:43498
I20260927 06:59:04.779693 18482 tablet_service.cc:1514] Processing DeleteTablet for tablet 7780bbf01ea24b54b161f153383a375b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:04 UTC) from {username='slave'} at 127.0.0.1:59478
I20260927 06:59:04.779765 18221 tablet_service.cc:1514] Processing DeleteTablet for tablet 7780bbf01ea24b54b161f153383a375b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:04 UTC) from {username='slave'} at 127.0.0.1:45216
I20260927 06:59:04.782773 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 18159
I20260927 06:59:04.791442 18591 tablet_replica.cc:330] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b: stopping tablet replica
I20260927 06:59:04.793161 18590 tablet_replica.cc:330] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9: stopping tablet replica
I20260927 06:59:04.793186 18591 raft_consensus.cc:2229] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:59:04.793320 18589 tablet_replica.cc:330] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a: stopping tablet replica
I20260927 06:59:04.794018 18591 raft_consensus.cc:2258] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:04.794552 18590 raft_consensus.cc:2229] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:04.795038 18590 raft_consensus.cc:2258] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:04.795123 18589 raft_consensus.cc:2229] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:04.795578 18589 raft_consensus.cc:2258] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:04.796559 18591 ts_tablet_manager.cc:1905] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:04.797164 18590 ts_tablet_manager.cc:1905] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:04.797729 18589 ts_tablet_manager.cc:1905] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:04.803738 18591 ts_tablet_manager.cc:1918] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:04.803948 18591 log.cc:1196] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/7780bbf01ea24b54b161f153383a375b
I20260927 06:59:04.804427 18589 ts_tablet_manager.cc:1918] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:04.804572 18589 log.cc:1196] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/7780bbf01ea24b54b161f153383a375b
I20260927 06:59:04.804567 18590 ts_tablet_manager.cc:1918] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:04.804610 18591 ts_tablet_manager.cc:1939] T 7780bbf01ea24b54b161f153383a375b P 1c44168caf1345319552d5e62e748e0b: Deleting consensus metadata
I20260927 06:59:04.804725 18590 log.cc:1196] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/7780bbf01ea24b54b161f153383a375b
I20260927 06:59:04.805078 18589 ts_tablet_manager.cc:1939] T 7780bbf01ea24b54b161f153383a375b P 214ff3d4d4cc46b982f1e4b28c6bca6a: Deleting consensus metadata
I20260927 06:59:04.805285 18590 ts_tablet_manager.cc:1939] T 7780bbf01ea24b54b161f153383a375b P da956d9bcba5409390340ecb28ac08f9: Deleting consensus metadata
I20260927 06:59:04.806646 18084 catalog_manager.cc:4841] TS 214ff3d4d4cc46b982f1e4b28c6bca6a (127.0.97.3:46823): tablet 7780bbf01ea24b54b161f153383a375b (table test-table [id=b5fecfea767945a5b8a2145700864032]) successfully deleted
I20260927 06:59:04.807006 18086 catalog_manager.cc:4841] TS 1c44168caf1345319552d5e62e748e0b (127.0.97.1:42885): tablet 7780bbf01ea24b54b161f153383a375b (table test-table [id=b5fecfea767945a5b8a2145700864032]) successfully deleted
I20260927 06:59:04.807283 18086 catalog_manager.cc:4841] TS da956d9bcba5409390340ecb28ac08f9 (127.0.97.2:32771): tablet 7780bbf01ea24b54b161f153383a375b (table test-table [id=b5fecfea767945a5b8a2145700864032]) successfully deleted
I20260927 06:59:04.812045 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:04.784138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45230 (local address 127.0.97.1:42885)
0927 06:59:04.784346 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:04.784355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:04.804319 (+ 19964us) server_negotiation.cc:407] Connection header received
0927 06:59:04.804837 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:04.804845 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:04.804970 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:04.805050 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:04.805717 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.806948 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.809031 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.809451 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.809545 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:04.810702 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:04.810710 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:04.810722 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:04.811270 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:04.811316 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:04.811334 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:04.811415 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:04.811584 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 06:59:04.811659 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 06:59:04.812688 18281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:04.911439 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 18159
I20260927 06:59:04.930411 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 18290
I20260927 06:59:04.937435 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:04.931254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.0.97.2:32771)
0927 06:59:04.931451 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:04.931457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:04.931480 (+ 23us) server_negotiation.cc:407] Connection header received
0927 06:59:04.931550 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:04.931560 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:04.931644 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:04.931755 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:04.932158 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.933013 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.934519 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:04.934914 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:04.935043 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:04.936135 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:04.936143 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:04.936156 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:04.936696 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:04.936740 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:04.936758 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:04.936896 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:04.937015 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 06:59:04.937118 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 06:59:04.938162 18412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:05.025094 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 18290
I20260927 06:59:05.041597 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 18420
I20260927 06:59:05.051590 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.046129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.0.97.3:46823)
0927 06:59:05.046287 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:05.046291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:05.046308 (+ 17us) server_negotiation.cc:407] Connection header received
0927 06:59:05.046439 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:05.046444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:05.046526 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:05.046618 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:05.046968 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.047694 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.048904 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.049284 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.049370 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.050385 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:05.050394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:05.050409 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:05.050987 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:05.051034 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:05.051052 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:05.051142 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:05.051251 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 06:59:05.051333 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 06:59:05.052291 18542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:05.148518 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 18420
I20260927 06:59:05.165042 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 18069
I20260927 06:59:05.166131 18130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:05.274504 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 18069
2026-09-27T06:59:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2642 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:05 dist-test-slave-f892 krb5kdc[18599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:05 dist-test-slave-f892 krb5kdc[18599](info): set up 2 sockets
Sep 27 06:59:05 dist-test-slave-f892 krb5kdc[18599](info): commencing operation
krb5kdc: starting...
W20260927 06:59:06.361972 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:06 dist-test-slave-f892 krb5kdc[18599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:59:06.597211 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40977
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:06.915267 18612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:06.915630 18612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:06.915807 18612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:06.923139 18612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:06.923275 18612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:06.923396 18612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:06.923503 18612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:06.923612 18612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:06.933172 18612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:06.934554 18612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:06.935863 18612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:06.947129 18620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:06.947245 18618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:06.947232 18617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:06.948202 18612 server_base.cc:1034] running on GCE node
I20260927 06:59:06.949002 18612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:06.950389 18612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:06.951732 18612 hybrid_clock.cc:648] HybridClock initialized: now 1790492346951691 us; error 43 us; skew 500 ppm
I20260927 06:59:06.952087 18612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:06 dist-test-slave-f892 krb5kdc[18599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:06.955128 18612 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:59:06.956553 18612 webserver.cc:460] Webserver started at http://127.0.97.62:44891/ using document root <none> and password file <none>
I20260927 06:59:06.957357 18612 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:06.957504 18612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:06.957799 18612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:06.960641 18612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "f93c21debf484001b07c289fbbf470ed"
format_stamp: "Formatted at 2026-09-27 06:59:06 on dist-test-slave-f892"
server_key: "c4b3f02682e75289215a0aa83ea2f88b"
server_key_iv: "63d610ae0ae2d56284852d2ef905ecee"
server_key_version: "encryptionkey@0"
I20260927 06:59:06.961386 18612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "f93c21debf484001b07c289fbbf470ed"
format_stamp: "Formatted at 2026-09-27 06:59:06 on dist-test-slave-f892"
server_key: "c4b3f02682e75289215a0aa83ea2f88b"
server_key_iv: "63d610ae0ae2d56284852d2ef905ecee"
server_key_version: "encryptionkey@0"
I20260927 06:59:06.966142 18612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20260927 06:59:06.969244 18627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:06.970494 18612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 06:59:06.970675 18612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "f93c21debf484001b07c289fbbf470ed"
format_stamp: "Formatted at 2026-09-27 06:59:06 on dist-test-slave-f892"
server_key: "c4b3f02682e75289215a0aa83ea2f88b"
server_key_iv: "63d610ae0ae2d56284852d2ef905ecee"
server_key_version: "encryptionkey@0"
I20260927 06:59:06.970870 18612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:06.985134 18612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:06.989177 18612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:06.989454 18612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:07.007220 18612 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40977
I20260927 06:59:07.007315 18678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40977 every 8 connection(s)
I20260927 06:59:07.008571 18612 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:07.011981 18679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:07.017870 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 18612
I20260927 06:59:07.018105 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:07.018622 388 external_mini_cluster.cc:1428] Setting key ee99da0ca8cd78a30b7020821488d2a1
I20260927 06:59:07.023139 18679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed: Bootstrap starting.
I20260927 06:59:07.026895 18679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:07.028303 18679 log.cc:824] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:07.031661 18679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed: No bootstrap required, opened a new log
Sep 27 06:59:07 dist-test-slave-f892 krb5kdc[18599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:59:07.039693 18679 raft_consensus.cc:348] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93c21debf484001b07c289fbbf470ed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40977 } }
I20260927 06:59:07.040072 18679 raft_consensus.cc:374] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:07.040195 18679 raft_consensus.cc:729] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93c21debf484001b07c289fbbf470ed, State: Initialized, Role: FOLLOWER
I20260927 06:59:07.040875 18679 consensus_queue.cc:260] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93c21debf484001b07c289fbbf470ed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40977 } }
I20260927 06:59:07.041157 18679 raft_consensus.cc:388] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:07.041306 18679 raft_consensus.cc:482] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:07.041488 18679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:07.042528 18682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.020537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43568 (local address 127.0.97.62:40977)
0927 06:59:07.021393 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:07.021405 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:07.021444 (+ 39us) server_negotiation.cc:407] Connection header received
0927 06:59:07.023169 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:07.023192 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:07.023590 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:07.023924 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:07.025660 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.027072 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.029021 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.029960 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.030157 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.034936 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:07.034946 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:07.034959 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:07.034995 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:07.038202 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.038913 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.038919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.038925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.039080 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.039452 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.039459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.039462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.039959 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:07.040262 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:07.040721 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 06:59:07.041140 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":454,"threads_started":1}
I20260927 06:59:07.044137 18679 raft_consensus.cc:504] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93c21debf484001b07c289fbbf470ed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40977 } }
I20260927 06:59:07.044616 18679 leader_election.cc:302] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f93c21debf484001b07c289fbbf470ed; no voters:
I20260927 06:59:07.045305 18679 leader_election.cc:288] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:07.045403 18684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:07.046211 18684 raft_consensus.cc:686] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed [term 1 LEADER]: Becoming Leader. State: Replica: f93c21debf484001b07c289fbbf470ed, State: Running, Role: LEADER
I20260927 06:59:07.046736 18684 consensus_queue.cc:237] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93c21debf484001b07c289fbbf470ed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40977 } }
I20260927 06:59:07.047447 18679 sys_catalog.cc:564] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:07.049437 18686 sys_catalog.cc:455] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader f93c21debf484001b07c289fbbf470ed. Latest consensus state: current_term: 1 leader_uuid: "f93c21debf484001b07c289fbbf470ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93c21debf484001b07c289fbbf470ed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40977 } } }
I20260927 06:59:07.049813 18686 sys_catalog.cc:458] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:07.049624 18685 sys_catalog.cc:455] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f93c21debf484001b07c289fbbf470ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93c21debf484001b07c289fbbf470ed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40977 } } }
I20260927 06:59:07.050091 18685 sys_catalog.cc:458] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:07.055351 18691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:07.059408 18691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:07.067546 18691 catalog_manager.cc:1347] Generated new cluster ID: 1f6c7b28fd4a41dbaa1c2b09a6b15a30
I20260927 06:59:07.067679 18691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:07.077903 18691 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:07.079053 18691 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:07.095888 18691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed: Generated new TSK 0
I20260927 06:59:07.096701 18691 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 06:59:07.165028 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40977
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:07.485690 18705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:07.485960 18705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:07.486116 18705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:07.493221 18705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:07.493320 18705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:07.493431 18705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:59:07.502888 18705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:07.504014 18705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:07.505198 18705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:07.518075 18710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:07.517993 18713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:07.518110 18711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:07.519111 18705 server_base.cc:1034] running on GCE node
I20260927 06:59:07.519759 18705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:07.520620 18705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:07.521831 18705 hybrid_clock.cc:648] HybridClock initialized: now 1790492347521786 us; error 42 us; skew 500 ppm
I20260927 06:59:07.522192 18705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:07 dist-test-slave-f892 krb5kdc[18599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:07.535609 18705 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:59:07.537091 18705 webserver.cc:460] Webserver started at http://127.0.97.1:45097/ using document root <none> and password file <none>
I20260927 06:59:07.537714 18705 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:07.537812 18705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:07.538159 18705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:07.540940 18705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "ef6322b7375f493dbc808285949c2efa"
format_stamp: "Formatted at 2026-09-27 06:59:07 on dist-test-slave-f892"
server_key: "9ca2ad401d66f771aa65c611cbffdebd"
server_key_iv: "238fd46ba8d8c8350bbe248f1ce17650"
server_key_version: "encryptionkey@0"
I20260927 06:59:07.541851 18705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ef6322b7375f493dbc808285949c2efa"
format_stamp: "Formatted at 2026-09-27 06:59:07 on dist-test-slave-f892"
server_key: "9ca2ad401d66f771aa65c611cbffdebd"
server_key_iv: "238fd46ba8d8c8350bbe248f1ce17650"
server_key_version: "encryptionkey@0"
I20260927 06:59:07.546837 18705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 06:59:07.550199 18720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:07.551493 18705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 06:59:07.551658 18705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "ef6322b7375f493dbc808285949c2efa"
format_stamp: "Formatted at 2026-09-27 06:59:07 on dist-test-slave-f892"
server_key: "9ca2ad401d66f771aa65c611cbffdebd"
server_key_iv: "238fd46ba8d8c8350bbe248f1ce17650"
server_key_version: "encryptionkey@0"
I20260927 06:59:07.551856 18705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:07.587925 18705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:07.592221 18705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:07.592525 18705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:07.593472 18705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:07.594885 18705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:07.594986 18705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:07.595141 18705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:07.595216 18705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:07.627959 18705 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40005
I20260927 06:59:07.628036 18832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40005 every 8 connection(s)
I20260927 06:59:07.629253 18705 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:59:07.637765 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 18705
I20260927 06:59:07.637948 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:59:07.638494 388 external_mini_cluster.cc:1428] Setting key b688876a374cdd5b804fec3be1d5f497
Sep 27 06:59:07 dist-test-slave-f892 krb5kdc[18599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:59:07.649973 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.631755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57987 (local address 127.0.97.62:40977)
0927 06:59:07.632198 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:07.632203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:07.633126 (+ 923us) server_negotiation.cc:407] Connection header received
0927 06:59:07.635009 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:07.635014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:07.635093 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:07.635188 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:07.636591 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.637295 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.639505 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.639956 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.640056 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.643865 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:07.643871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:07.643874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:07.643905 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:07.646549 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.647409 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.647417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.647423 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.647578 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.648124 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.648127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.648129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.648401 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:07.648512 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:07.649523 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0927 06:59:07.649701 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":335,"thread_start_us":217,"threads_started":1}
I20260927 06:59:07.651572 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.632292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40977 (local address 127.0.97.1:57987)
0927 06:59:07.632966 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:07.633003 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:07.634765 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:07.635400 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:07.635408 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:07.635932 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:07.636446 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.636463 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.637491 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.637497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.639373 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.639385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.640210 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.640218 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.640499 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.641225 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:07.641253 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:07.643636 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:07.646706 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:07.646715 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:07.646732 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:07.647226 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:07.647735 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:07.647741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:07.647747 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:07.648002 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:07.648524 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:07.648534 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:07.648996 (+ 462us) client_negotiation.cc:769] Sending connection context
0927 06:59:07.649414 (+ 418us) client_negotiation.cc:240] Negotiation successful
0927 06:59:07.650037 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":281,"threads_started":1}
I20260927 06:59:07.653290 18833 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40977
I20260927 06:59:07.653601 18833 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:07.654376 18833 heartbeater.cc:511] Master 127.0.97.62:40977 requested a full tablet report, sending...
I20260927 06:59:07.656467 18644 ts_manager.cc:194] Registered new tserver with Master: ef6322b7375f493dbc808285949c2efa (127.0.97.1:40005)
I20260927 06:59:07.657929 18644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57987
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 06:59:07.694694 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40977
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:08.003597 18839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:08.003841 18839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:08.003979 18839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:08.010845 18839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:08.010986 18839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:08.011165 18839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:59:08.020632 18839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:08.021836 18839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:08.023144 18839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:08.036047 18845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:08.036194 18844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:08.036226 18847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:08.036903 18839 server_base.cc:1034] running on GCE node
I20260927 06:59:08.037467 18839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:08.038504 18839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:08.039729 18839 hybrid_clock.cc:648] HybridClock initialized: now 1790492348039663 us; error 50 us; skew 500 ppm
I20260927 06:59:08.040028 18839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:08 dist-test-slave-f892 krb5kdc[18599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:08.042955 18839 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:59:08.044293 18839 webserver.cc:460] Webserver started at http://127.0.97.2:45511/ using document root <none> and password file <none>
I20260927 06:59:08.044950 18839 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:08.045090 18839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:08.045394 18839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:08.048373 18839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "7dd6d766780441a5a6ff14d107d8c199"
format_stamp: "Formatted at 2026-09-27 06:59:08 on dist-test-slave-f892"
server_key: "19ad53a38110831ef9214410cc36b4bf"
server_key_iv: "36df1053f4babcfd9ffbdb35e8d8015b"
server_key_version: "encryptionkey@0"
I20260927 06:59:08.049127 18839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7dd6d766780441a5a6ff14d107d8c199"
format_stamp: "Formatted at 2026-09-27 06:59:08 on dist-test-slave-f892"
server_key: "19ad53a38110831ef9214410cc36b4bf"
server_key_iv: "36df1053f4babcfd9ffbdb35e8d8015b"
server_key_version: "encryptionkey@0"
I20260927 06:59:08.054270 18839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 06:59:08.057605 18854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:08.058847 18839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:59:08.059082 18839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "7dd6d766780441a5a6ff14d107d8c199"
format_stamp: "Formatted at 2026-09-27 06:59:08 on dist-test-slave-f892"
server_key: "19ad53a38110831ef9214410cc36b4bf"
server_key_iv: "36df1053f4babcfd9ffbdb35e8d8015b"
server_key_version: "encryptionkey@0"
I20260927 06:59:08.059309 18839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:08.074569 18839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:08.078132 18839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:08.078401 18839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:08.079298 18839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:08.080511 18839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:08.080610 18839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:08.080755 18839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:08.080837 18839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:08.112674 18839 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41755
I20260927 06:59:08.112772 18966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41755 every 8 connection(s)
I20260927 06:59:08.114058 18839 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:59:08.115820 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 18839
I20260927 06:59:08.115998 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:59:08.116472 388 external_mini_cluster.cc:1428] Setting key 33877989ab3aa934d30b6e3ae61c9e95
Sep 27 06:59:08 dist-test-slave-f892 krb5kdc[18599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:59:08.135303 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.116622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39165 (local address 127.0.97.62:40977)
0927 06:59:08.116832 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.116838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.117953 (+ 1115us) server_negotiation.cc:407] Connection header received
0927 06:59:08.119797 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.119803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.119883 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.119997 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:08.121412 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.122196 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.123677 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.124111 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.124245 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.128330 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:08.128339 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:08.128347 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:08.128392 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:08.132012 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:08.132738 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:08.132746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:08.132752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:08.132907 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:08.133308 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:08.133315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:08.133322 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:08.133699 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:08.133855 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.134580 (+ 725us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.134843 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 06:59:08.135915 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.117110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40977 (local address 127.0.97.2:39165)
0927 06:59:08.117808 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:08.117842 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:08.119497 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:08.120230 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:08.120238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:08.120709 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:08.121229 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.121246 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.122339 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.122344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.123529 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.123540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.124328 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.124335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.124637 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.125423 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:08.125457 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:08.128100 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:08.132164 (+ 4064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:08.132170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:08.132183 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:08.132603 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:08.133016 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:08.133019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:08.133022 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:08.133193 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:08.133822 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:08.133828 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:08.134205 (+ 377us) client_negotiation.cc:769] Sending connection context
0927 06:59:08.134444 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 06:59:08.134832 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":317,"threads_started":1}
I20260927 06:59:08.137426 18967 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40977
I20260927 06:59:08.137722 18967 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:08.138484 18967 heartbeater.cc:511] Master 127.0.97.62:40977 requested a full tablet report, sending...
I20260927 06:59:08.139981 18644 ts_manager.cc:194] Registered new tserver with Master: 7dd6d766780441a5a6ff14d107d8c199 (127.0.97.2:41755)
I20260927 06:59:08.141165 18644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39165
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 06:59:08.173341 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40977
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:08.489039 18972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:08.489280 18972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:08.489393 18972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:08.496389 18972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:08.496482 18972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:08.496693 18972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:59:08.505751 18972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:08.506896 18972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:08.508118 18972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:08.520823 18977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:08.520754 18978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:08.520785 18980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:08.522199 18972 server_base.cc:1034] running on GCE node
I20260927 06:59:08.522843 18972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:08.523787 18972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:08.525019 18972 hybrid_clock.cc:648] HybridClock initialized: now 1790492348524980 us; error 49 us; skew 500 ppm
I20260927 06:59:08.525362 18972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:08 dist-test-slave-f892 krb5kdc[18599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:08.528415 18972 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:59:08.529901 18972 webserver.cc:460] Webserver started at http://127.0.97.3:36569/ using document root <none> and password file <none>
I20260927 06:59:08.530648 18972 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:08.530798 18972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:08.531107 18972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:08.533905 18972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "5900e8db21074d428790ac2ea56279d3"
format_stamp: "Formatted at 2026-09-27 06:59:08 on dist-test-slave-f892"
server_key: "9d7b01641ff33c8b8c35f4daa6d738c9"
server_key_iv: "7b5342a2cef9427a9210ee98b4aa64a9"
server_key_version: "encryptionkey@0"
I20260927 06:59:08.534780 18972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5900e8db21074d428790ac2ea56279d3"
format_stamp: "Formatted at 2026-09-27 06:59:08 on dist-test-slave-f892"
server_key: "9d7b01641ff33c8b8c35f4daa6d738c9"
server_key_iv: "7b5342a2cef9427a9210ee98b4aa64a9"
server_key_version: "encryptionkey@0"
I20260927 06:59:08.539788 18972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 06:59:08.543104 18987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:08.544399 18972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 06:59:08.544572 18972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "5900e8db21074d428790ac2ea56279d3"
format_stamp: "Formatted at 2026-09-27 06:59:08 on dist-test-slave-f892"
server_key: "9d7b01641ff33c8b8c35f4daa6d738c9"
server_key_iv: "7b5342a2cef9427a9210ee98b4aa64a9"
server_key_version: "encryptionkey@0"
I20260927 06:59:08.544761 18972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:08.563445 18972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:08.567198 18972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:08.567471 18972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:08.568395 18972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:08.569662 18972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:08.569763 18972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:08.569916 18972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:08.569998 18972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:08.602574 18972 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33893
I20260927 06:59:08.602622 19099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33893 every 8 connection(s)
I20260927 06:59:08.603778 18972 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:59:08.604462 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 18972
I20260927 06:59:08.604619 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:59:08.605166 388 external_mini_cluster.cc:1428] Setting key b7512b4e35d916a1a61fdef08cfd12e3
Sep 27 06:59:08 dist-test-slave-f892 krb5kdc[18599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:59:08.624279 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.606816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32885 (local address 127.0.97.62:40977)
0927 06:59:08.607009 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.607014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.608345 (+ 1331us) server_negotiation.cc:407] Connection header received
0927 06:59:08.610146 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.610151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.610234 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.610354 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:08.611851 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.613056 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.614732 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.615156 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.615289 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.618740 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:08.618747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:08.618753 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:08.618791 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:08.621643 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:08.622386 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:08.622390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:08.622392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:08.622511 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:08.622868 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:08.622871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:08.622873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:08.623144 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:08.623257 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.623888 (+ 631us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.624037 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 06:59:08.625183 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.607343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40977 (local address 127.0.97.3:32885)
0927 06:59:08.608178 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:08.608235 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:08.609874 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:08.610580 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:08.610587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:08.611064 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:08.611588 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.611603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.613208 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.613214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.614585 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.614597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.615399 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.615407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.615636 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.616323 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:08.616345 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:08.618487 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:08.621766 (+ 3279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:08.621773 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:08.621785 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:08.622266 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:08.622602 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:08.622605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:08.622607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:08.622774 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:08.623261 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:08.623266 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:08.623549 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 06:59:08.623790 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 06:59:08.624131 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":311,"threads_started":1}
I20260927 06:59:08.626580 19100 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40977
I20260927 06:59:08.626839 19100 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:08.627457 19100 heartbeater.cc:511] Master 127.0.97.62:40977 requested a full tablet report, sending...
I20260927 06:59:08.628868 18644 ts_manager.cc:194] Registered new tserver with Master: 5900e8db21074d428790ac2ea56279d3 (127.0.97.3:33893)
I20260927 06:59:08.629927 18644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:32885
I20260927 06:59:08.632444 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:59:08.648412 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.636588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.0.97.62:40977)
0927 06:59:08.636813 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.636821 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.637105 (+ 284us) server_negotiation.cc:407] Connection header received
0927 06:59:08.637212 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.637215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.637267 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.637340 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:08.638643 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.639336 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.640647 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.641027 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.641125 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.643669 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:08.643673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:08.643676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:08.643702 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:08.645928 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:08.646567 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:08.646570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:08.646572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:08.646695 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:08.647028 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:08.647031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:08.647033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:08.647284 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:08.647463 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.647943 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.648115 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 06:59:08.652141 18644 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:59:08.659117 18644 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:59:08.660787 18833 heartbeater.cc:503] Master 127.0.97.62:40977 was elected leader, sending a full tablet report...
I20260927 06:59:08.681507 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.672378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34070 (local address 127.0.97.1:40005)
0927 06:59:08.673617 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.673623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.673644 (+ 21us) server_negotiation.cc:407] Connection header received
0927 06:59:08.673709 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.673714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.673935 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.674145 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:08.674844 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.676480 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.678435 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.679985 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.680113 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.680334 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.680992 (+ 658us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.681154 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1022,"thread_start_us":886,"threads_started":1}
I20260927 06:59:08.681542 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.671681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40005 (local address 127.0.0.1:34070)
0927 06:59:08.672932 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:08.672947 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:08.673052 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:08.674154 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:08.674161 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:08.674187 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:08.674601 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.674615 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.676645 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.676649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.678284 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.678295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.680471 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.680478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.680722 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.680777 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 06:59:08.680922 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 06:59:08.681096 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"mutex_wait_us":23,"thread_start_us":3269,"threads_started":1}
I20260927 06:59:08.683597 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.671458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.0.97.3:33893)
0927 06:59:08.675697 (+ 4239us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.675706 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.675738 (+ 32us) server_negotiation.cc:407] Connection header received
0927 06:59:08.675825 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.675834 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.676041 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.676935 (+ 894us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:08.677151 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.678721 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.681343 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.682532 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.682706 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.682934 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.683105 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.683270 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4056,"thread_start_us":3871,"threads_started":1}
I20260927 06:59:08.684319 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.671106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33893 (local address 127.0.0.1:37324)
0927 06:59:08.672032 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:08.672073 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:08.672226 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:08.676290 (+ 4064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:08.676295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:08.676323 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:08.676778 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.676790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.678867 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.678872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.681160 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.681173 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.682764 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.682770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.682928 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.682960 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 06:59:08.683055 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 06:59:08.683187 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":62,"thread_start_us":498,"threads_started":1}
I20260927 06:59:08.685225 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.672835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55666 (local address 127.0.97.2:41755)
0927 06:59:08.674822 (+ 1987us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.674829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.675497 (+ 668us) server_negotiation.cc:407] Connection header received
0927 06:59:08.675619 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.675626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.675814 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.675985 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:08.677135 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.678710 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.681104 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.683588 (+ 2484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.684530 (+ 942us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.684702 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.684770 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.684910 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1847,"thread_start_us":1706,"threads_started":1}
I20260927 06:59:08.685657 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.672421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41755 (local address 127.0.0.1:55666)
0927 06:59:08.675217 (+ 2796us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:08.675233 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:08.675357 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:08.676181 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:08.676185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:08.676208 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:08.676958 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.676973 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.678911 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.678918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.680923 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.680938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.683935 (+ 2997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.683940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.684183 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.684220 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 06:59:08.684295 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 06:59:08.685251 (+ 956us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2688,"mutex_wait_us":274,"spinlock_wait_cycles":896,"thread_start_us":264,"threads_started":1}
I20260927 06:59:08.685374 18768 tablet_service.cc:1467] Processing CreateTablet for tablet f15410d18cb4430d873224bec77642dc (DEFAULT_TABLE table=test-table [id=af39509b0ec84c69b3d7cf2b34b71e74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:08.686919 18768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15410d18cb4430d873224bec77642dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:08.687803 19035 tablet_service.cc:1467] Processing CreateTablet for tablet f15410d18cb4430d873224bec77642dc (DEFAULT_TABLE table=test-table [id=af39509b0ec84c69b3d7cf2b34b71e74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:08.688268 18902 tablet_service.cc:1467] Processing CreateTablet for tablet f15410d18cb4430d873224bec77642dc (DEFAULT_TABLE table=test-table [id=af39509b0ec84c69b3d7cf2b34b71e74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:08.689193 19035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15410d18cb4430d873224bec77642dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:08.689533 18902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15410d18cb4430d873224bec77642dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:08.699965 19118 tablet_bootstrap.cc:492] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199: Bootstrap starting.
I20260927 06:59:08.703549 19120 tablet_bootstrap.cc:492] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3: Bootstrap starting.
I20260927 06:59:08.704646 19118 tablet_bootstrap.cc:654] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:08.705894 19118 log.cc:824] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:08.707109 19120 tablet_bootstrap.cc:654] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:08.707253 19119 tablet_bootstrap.cc:492] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa: Bootstrap starting.
I20260927 06:59:08.708318 19120 log.cc:824] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:08.709170 19118 tablet_bootstrap.cc:492] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199: No bootstrap required, opened a new log
I20260927 06:59:08.709419 19118 ts_tablet_manager.cc:1397] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 06:59:08.711450 19119 tablet_bootstrap.cc:654] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:08.711633 19120 tablet_bootstrap.cc:492] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3: No bootstrap required, opened a new log
I20260927 06:59:08.711884 19120 ts_tablet_manager.cc:1397] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 06:59:08.712464 19119 log.cc:824] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:08.715447 19119 tablet_bootstrap.cc:492] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa: No bootstrap required, opened a new log
I20260927 06:59:08.715718 19119 ts_tablet_manager.cc:1397] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260927 06:59:08.720438 19118 raft_consensus.cc:348] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5900e8db21074d428790ac2ea56279d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33893 } } peers { permanent_uuid: "7dd6d766780441a5a6ff14d107d8c199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41755 } } peers { permanent_uuid: "ef6322b7375f493dbc808285949c2efa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40005 } }
I20260927 06:59:08.720934 19118 raft_consensus.cc:374] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:08.721024 19118 raft_consensus.cc:729] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dd6d766780441a5a6ff14d107d8c199, State: Initialized, Role: FOLLOWER
I20260927 06:59:08.721758 19118 consensus_queue.cc:260] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5900e8db21074d428790ac2ea56279d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33893 } } peers { permanent_uuid: "7dd6d766780441a5a6ff14d107d8c199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41755 } } peers { permanent_uuid: "ef6322b7375f493dbc808285949c2efa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40005 } }
I20260927 06:59:08.722494 19120 raft_consensus.cc:348] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5900e8db21074d428790ac2ea56279d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33893 } } peers { permanent_uuid: "7dd6d766780441a5a6ff14d107d8c199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41755 } } peers { permanent_uuid: "ef6322b7375f493dbc808285949c2efa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40005 } }
I20260927 06:59:08.722971 19120 raft_consensus.cc:374] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:08.723076 19120 raft_consensus.cc:729] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5900e8db21074d428790ac2ea56279d3, State: Initialized, Role: FOLLOWER
I20260927 06:59:08.723222 19119 raft_consensus.cc:348] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5900e8db21074d428790ac2ea56279d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33893 } } peers { permanent_uuid: "7dd6d766780441a5a6ff14d107d8c199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41755 } } peers { permanent_uuid: "ef6322b7375f493dbc808285949c2efa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40005 } }
I20260927 06:59:08.723508 18967 heartbeater.cc:503] Master 127.0.97.62:40977 was elected leader, sending a full tablet report...
I20260927 06:59:08.723646 19119 raft_consensus.cc:374] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:08.723730 19119 raft_consensus.cc:729] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef6322b7375f493dbc808285949c2efa, State: Initialized, Role: FOLLOWER
I20260927 06:59:08.723654 19120 consensus_queue.cc:260] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5900e8db21074d428790ac2ea56279d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33893 } } peers { permanent_uuid: "7dd6d766780441a5a6ff14d107d8c199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41755 } } peers { permanent_uuid: "ef6322b7375f493dbc808285949c2efa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40005 } }
I20260927 06:59:08.724301 19118 ts_tablet_manager.cc:1428] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260927 06:59:08.724368 19119 consensus_queue.cc:260] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5900e8db21074d428790ac2ea56279d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33893 } } peers { permanent_uuid: "7dd6d766780441a5a6ff14d107d8c199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41755 } } peers { permanent_uuid: "ef6322b7375f493dbc808285949c2efa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40005 } }
I20260927 06:59:08.725241 19100 heartbeater.cc:503] Master 127.0.97.62:40977 was elected leader, sending a full tablet report...
I20260927 06:59:08.726688 19119 ts_tablet_manager.cc:1428] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260927 06:59:08.727319 19120 ts_tablet_manager.cc:1428] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260927 06:59:08.805228 19124 raft_consensus.cc:482] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:59:08.805505 19124 raft_consensus.cc:504] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5900e8db21074d428790ac2ea56279d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33893 } } peers { permanent_uuid: "7dd6d766780441a5a6ff14d107d8c199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41755 } } peers { permanent_uuid: "ef6322b7375f493dbc808285949c2efa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40005 } }
I20260927 06:59:08.807118 19124 leader_election.cc:288] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5900e8db21074d428790ac2ea56279d3 (127.0.97.3:33893), ef6322b7375f493dbc808285949c2efa (127.0.97.1:40005)
I20260927 06:59:08.813237 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.807538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54691 (local address 127.0.97.1:40005)
0927 06:59:08.807803 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.807808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.808141 (+ 333us) server_negotiation.cc:407] Connection header received
0927 06:59:08.808384 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.808393 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.808453 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.808580 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:08.809089 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.809806 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.811541 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.812293 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.812456 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.812565 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.812853 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.812979 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260927 06:59:08.813406 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.807366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40005 (local address 127.0.97.2:54691)
0927 06:59:08.808022 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:08.808036 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:08.808221 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:08.808583 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:08.808588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:08.808615 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:08.808944 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.808952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.809936 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.809943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.811388 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.811400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.812518 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.812524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.812681 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.812705 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 06:59:08.812817 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 06:59:08.812953 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":1,"thread_start_us":325,"threads_started":1}
I20260927 06:59:08.813688 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.807519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58127 (local address 127.0.97.3:33893)
0927 06:59:08.807781 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.807790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.808636 (+ 846us) server_negotiation.cc:407] Connection header received
0927 06:59:08.808850 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.808861 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.808921 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.809016 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:08.809522 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.810304 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.811963 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.812726 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.812877 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.812986 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.813294 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.813428 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260927 06:59:08.813800 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.807366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33893 (local address 127.0.97.2:58127)
0927 06:59:08.808464 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:08.808482 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:08.808692 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:08.809030 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:08.809034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:08.809056 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:08.809371 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.809379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.810435 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.810442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.811797 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.811809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.812938 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.812944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.813127 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.813151 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 06:59:08.813268 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 06:59:08.813408 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"mutex_wait_us":453,"thread_start_us":328,"threads_started":1}
I20260927 06:59:08.814662 18788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15410d18cb4430d873224bec77642dc" candidate_uuid: "7dd6d766780441a5a6ff14d107d8c199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef6322b7375f493dbc808285949c2efa" is_pre_election: true
I20260927 06:59:08.814699 19055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15410d18cb4430d873224bec77642dc" candidate_uuid: "7dd6d766780441a5a6ff14d107d8c199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5900e8db21074d428790ac2ea56279d3" is_pre_election: true
I20260927 06:59:08.815228 18788 raft_consensus.cc:2454] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7dd6d766780441a5a6ff14d107d8c199 in term 0.
I20260927 06:59:08.815274 19055 raft_consensus.cc:2454] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7dd6d766780441a5a6ff14d107d8c199 in term 0.
I20260927 06:59:08.815959 18856 leader_election.cc:302] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5900e8db21074d428790ac2ea56279d3, 7dd6d766780441a5a6ff14d107d8c199, ef6322b7375f493dbc808285949c2efa; no voters:
I20260927 06:59:08.816370 19124 raft_consensus.cc:2781] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:59:08.816540 19124 raft_consensus.cc:482] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:59:08.816654 19124 raft_consensus.cc:3037] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:08.818910 19124 raft_consensus.cc:504] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5900e8db21074d428790ac2ea56279d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33893 } } peers { permanent_uuid: "7dd6d766780441a5a6ff14d107d8c199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41755 } } peers { permanent_uuid: "ef6322b7375f493dbc808285949c2efa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40005 } }
I20260927 06:59:08.819487 19124 leader_election.cc:288] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [CANDIDATE]: Term 1 election: Requested vote from peers 5900e8db21074d428790ac2ea56279d3 (127.0.97.3:33893), ef6322b7375f493dbc808285949c2efa (127.0.97.1:40005)
I20260927 06:59:08.819954 19055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15410d18cb4430d873224bec77642dc" candidate_uuid: "7dd6d766780441a5a6ff14d107d8c199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5900e8db21074d428790ac2ea56279d3"
I20260927 06:59:08.819962 18788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15410d18cb4430d873224bec77642dc" candidate_uuid: "7dd6d766780441a5a6ff14d107d8c199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef6322b7375f493dbc808285949c2efa"
I20260927 06:59:08.820147 19055 raft_consensus.cc:3037] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:08.820175 18788 raft_consensus.cc:3037] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:08.822335 18788 raft_consensus.cc:2454] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7dd6d766780441a5a6ff14d107d8c199 in term 1.
I20260927 06:59:08.822355 19055 raft_consensus.cc:2454] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7dd6d766780441a5a6ff14d107d8c199 in term 1.
I20260927 06:59:08.822737 18856 leader_election.cc:302] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7dd6d766780441a5a6ff14d107d8c199, ef6322b7375f493dbc808285949c2efa; no voters:
I20260927 06:59:08.823006 19124 raft_consensus.cc:2781] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:08.823433 19124 raft_consensus.cc:686] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [term 1 LEADER]: Becoming Leader. State: Replica: 7dd6d766780441a5a6ff14d107d8c199, State: Running, Role: LEADER
I20260927 06:59:08.823904 19124 consensus_queue.cc:237] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5900e8db21074d428790ac2ea56279d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33893 } } peers { permanent_uuid: "7dd6d766780441a5a6ff14d107d8c199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41755 } } peers { permanent_uuid: "ef6322b7375f493dbc808285949c2efa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40005 } }
I20260927 06:59:08.828094 18644 catalog_manager.cc:5495] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7dd6d766780441a5a6ff14d107d8c199 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "7dd6d766780441a5a6ff14d107d8c199" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5900e8db21074d428790ac2ea56279d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7dd6d766780441a5a6ff14d107d8c199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6322b7375f493dbc808285949c2efa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40005 } health_report { overall_health: UNKNOWN } } }
W20260927 06:59:08.856133 19101 tablet.cc:2369] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:59:08.866078 18968 tablet.cc:2369] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:59:08.873422 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.868469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.0.97.2:41755)
0927 06:59:08.868673 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.868677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.868695 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:59:08.868859 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.868862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.868937 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.869049 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:08.869377 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.870152 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.871436 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.871839 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.871926 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.872381 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:08.872840 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:08.872965 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.873078 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.873142 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 06:59:08.881596 18834 tablet.cc:2369] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:59:08.882691 19055 raft_consensus.cc:1261] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3 [term 1 FOLLOWER]: Refusing update from remote peer 7dd6d766780441a5a6ff14d107d8c199: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:08.883631 19124 consensus_queue.cc:1035] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5900e8db21074d428790ac2ea56279d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:08.884351 18788 raft_consensus.cc:1261] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa [term 1 FOLLOWER]: Refusing update from remote peer 7dd6d766780441a5a6ff14d107d8c199: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:08.885519 19124 consensus_queue.cc:1035] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6322b7375f493dbc808285949c2efa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:08.903084 19133 mvcc.cc:205] Tried to move back new op lower bound from 7333856661011943424 to 7333856660786913280. Current Snapshot: MvccSnapshot[applied={T|T < 7333856661011943424}]
I20260927 06:59:08.918560 19134 mvcc.cc:205] Tried to move back new op lower bound from 7333856661011943424 to 7333856660786913280. Current Snapshot: MvccSnapshot[applied={T|T < 7333856661011943424 or (T in {7333856661011943424})}]
I20260927 06:59:08.923811 18643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:08.924058 18643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:08.927366 18643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f93c21debf484001b07c289fbbf470ed: Sending DeleteTablet for 3 replicas of tablet f15410d18cb4430d873224bec77642dc
I20260927 06:59:08.928463 19035 tablet_service.cc:1514] Processing DeleteTablet for tablet f15410d18cb4430d873224bec77642dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37324
I20260927 06:59:08.928536 18902 tablet_service.cc:1514] Processing DeleteTablet for tablet f15410d18cb4430d873224bec77642dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55666
I20260927 06:59:08.929167 18768 tablet_service.cc:1514] Processing DeleteTablet for tablet f15410d18cb4430d873224bec77642dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34070
I20260927 06:59:08.931318 19143 tablet_replica.cc:330] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa: stopping tablet replica
I20260927 06:59:08.931722 19143 raft_consensus.cc:2229] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:08.932196 19143 raft_consensus.cc:2258] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:08.932341 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 18705
I20260927 06:59:08.934240 19143 ts_tablet_manager.cc:1905] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:08.941390 19143 ts_tablet_manager.cc:1918] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:08.941550 19143 log.cc:1196] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/f15410d18cb4430d873224bec77642dc
I20260927 06:59:08.942102 19143 ts_tablet_manager.cc:1939] T f15410d18cb4430d873224bec77642dc P ef6322b7375f493dbc808285949c2efa: Deleting consensus metadata
I20260927 06:59:08.943892 18629 catalog_manager.cc:4841] TS ef6322b7375f493dbc808285949c2efa (127.0.97.1:40005): tablet f15410d18cb4430d873224bec77642dc (table test-table [id=af39509b0ec84c69b3d7cf2b34b71e74]) successfully deleted
I20260927 06:59:08.948995 19141 tablet_replica.cc:330] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199: stopping tablet replica
I20260927 06:59:08.949491 19141 raft_consensus.cc:2229] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:59:08.949795 19142 tablet_replica.cc:330] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3: stopping tablet replica
I20260927 06:59:08.950219 19142 raft_consensus.cc:2229] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:08.950350 19141 raft_consensus.cc:2258] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:08.950783 19142 raft_consensus.cc:2258] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:08.952586 19141 ts_tablet_manager.cc:1905] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 06:59:08 dist-test-slave-f892 krb5kdc[18599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:59:08.955703 19142 ts_tablet_manager.cc:1905] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:08.959707 19141 ts_tablet_manager.cc:1918] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:08.959851 19141 log.cc:1196] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/f15410d18cb4430d873224bec77642dc
I20260927 06:59:08.960407 19141 ts_tablet_manager.cc:1939] T f15410d18cb4430d873224bec77642dc P 7dd6d766780441a5a6ff14d107d8c199: Deleting consensus metadata
I20260927 06:59:08.961984 18631 catalog_manager.cc:4841] TS 7dd6d766780441a5a6ff14d107d8c199 (127.0.97.2:41755): tablet f15410d18cb4430d873224bec77642dc (table test-table [id=af39509b0ec84c69b3d7cf2b34b71e74]) successfully deleted
I20260927 06:59:08.962872 19142 ts_tablet_manager.cc:1918] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:08.963057 19142 log.cc:1196] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/f15410d18cb4430d873224bec77642dc
I20260927 06:59:08.963572 19142 ts_tablet_manager.cc:1939] T f15410d18cb4430d873224bec77642dc P 5900e8db21074d428790ac2ea56279d3: Deleting consensus metadata
I20260927 06:59:08.965142 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.934365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.0.97.1:40005)
0927 06:59:08.934536 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.934543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.937985 (+ 3442us) server_negotiation.cc:407] Connection header received
0927 06:59:08.938950 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.938958 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.939078 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.939179 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:08.943762 (+ 4583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.945258 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.947991 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.948670 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.948804 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.956581 (+ 7777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:08.956596 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:08.956613 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:08.956668 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:08.961582 (+ 4914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:08.962527 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:08.962536 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:08.962538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:08.962714 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:08.963247 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:08.963256 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:08.963261 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:08.963783 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:08.964077 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.964384 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.964588 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 06:59:08.965135 18631 catalog_manager.cc:4841] TS 5900e8db21074d428790ac2ea56279d3 (127.0.97.3:33893): tablet f15410d18cb4430d873224bec77642dc (table test-table [id=af39509b0ec84c69b3d7cf2b34b71e74]) successfully deleted
I20260927 06:59:08.965759 18828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:09.116567 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 18705
I20260927 06:59:09.140373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 18839
Sep 27 06:59:09 dist-test-slave-f892 krb5kdc[18599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:59:09.159664 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:09.141293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55684 (local address 127.0.97.2:41755)
0927 06:59:09.141482 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:09.141489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:09.141509 (+ 20us) server_negotiation.cc:407] Connection header received
0927 06:59:09.141663 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:09.141668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:09.141752 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:09.141868 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:09.143594 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.144819 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.146697 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.147390 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.147523 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:09.152897 (+ 5374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:09.152909 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:09.152925 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:09.152967 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:09.156790 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:09.157564 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:09.157571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:09.157573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:09.157696 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:09.158188 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:09.158192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:09.158194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:09.158517 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:09.158748 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:09.159105 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 06:59:09.159306 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 06:59:09.160388 18962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:09.274539 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 18839
I20260927 06:59:09.292963 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 18972
Sep 27 06:59:09 dist-test-slave-f892 krb5kdc[18599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 06:59:09.308106 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:09.293649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.0.97.3:33893)
0927 06:59:09.293853 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:09.293861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:09.293882 (+ 21us) server_negotiation.cc:407] Connection header received
0927 06:59:09.294148 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:09.294153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:09.294228 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:09.294347 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:09.295665 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.296374 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.297669 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.298123 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.298262 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:09.302541 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:09.302554 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:09.302569 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:09.302600 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:09.305187 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:09.305798 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:09.305814 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:09.305817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:09.305949 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:09.306591 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:09.306596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:09.306599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:09.306937 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:09.307239 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:09.307533 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 06:59:09.307796 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 06:59:09.308737 19095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:09.396817 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 18972
I20260927 06:59:09.413532 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 18612
I20260927 06:59:09.414659 18674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:09.524583 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 18612
2026-09-27T06:59:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4251 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T06:59:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:09Z Disabled control of system clock
I20260927 06:59:09.597245 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38059
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 06:59:09.916936 19152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:09.917227 19152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:09.917306 19152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:09.924294 19152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:09.924400 19152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:09.924458 19152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:09.924506 19152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:09.924543 19152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:09.933449 19152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38059
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:09.934654 19152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:09.936451 19152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:09.947767 19158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:09.947779 19160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:09.947779 19157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:09.948583 19152 server_base.cc:1034] running on GCE node
I20260927 06:59:09.949235 19152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:09.950481 19152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:09.951676 19152 hybrid_clock.cc:648] HybridClock initialized: now 1790492349951640 us; error 26 us; skew 500 ppm
I20260927 06:59:09.952013 19152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:59:09.954205 19152 webserver.cc:460] Webserver started at http://127.0.97.62:38525/ using document root <none> and password file <none>
I20260927 06:59:09.954830 19152 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:09.954941 19152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:09.955259 19152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:09.958141 19152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "fcdbb3b90016455eb3cb05520b6aa2dc"
format_stamp: "Formatted at 2026-09-27 06:59:09 on dist-test-slave-f892"
server_key: "96db43854c261cdad0000892c99c53bd"
server_key_iv: "9a6d1c1276d8d73fdc9c7cda1326451b"
server_key_version: "encryptionkey@0"
I20260927 06:59:09.958913 19152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "fcdbb3b90016455eb3cb05520b6aa2dc"
format_stamp: "Formatted at 2026-09-27 06:59:09 on dist-test-slave-f892"
server_key: "96db43854c261cdad0000892c99c53bd"
server_key_iv: "9a6d1c1276d8d73fdc9c7cda1326451b"
server_key_version: "encryptionkey@0"
I20260927 06:59:09.963641 19152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 06:59:09.966933 19166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:09.968235 19152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 06:59:09.968454 19152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "fcdbb3b90016455eb3cb05520b6aa2dc"
format_stamp: "Formatted at 2026-09-27 06:59:09 on dist-test-slave-f892"
server_key: "96db43854c261cdad0000892c99c53bd"
server_key_iv: "9a6d1c1276d8d73fdc9c7cda1326451b"
server_key_version: "encryptionkey@0"
I20260927 06:59:09.968679 19152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:09.998646 19152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:09.999480 19152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:09.999786 19152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:10.019564 19152 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38059
I20260927 06:59:10.019608 19217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38059 every 8 connection(s)
I20260927 06:59:10.020893 19152 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:10.024501 19218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:10.026757 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 19152
I20260927 06:59:10.026940 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:10.027462 388 external_mini_cluster.cc:1428] Setting key bcf169af660c36f0fa2a22b8e3b67997
I20260927 06:59:10.036110 19218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc: Bootstrap starting.
I20260927 06:59:10.039287 19218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:10.040262 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.029349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.0.97.62:38059)
0927 06:59:10.030353 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:10.030362 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:10.030401 (+ 39us) server_negotiation.cc:407] Connection header received
0927 06:59:10.032051 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:10.032071 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:10.032482 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:10.032830 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:10.033383 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.034735 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.036015 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.036737 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.036930 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.038128 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:10.038136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:10.038148 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:10.038634 (+ 486us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:10.038717 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:10.038737 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:10.038865 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:10.039079 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 06:59:10.039357 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":342,"threads_started":1}
I20260927 06:59:10.040310 19218 log.cc:824] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:10.043090 19218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc: No bootstrap required, opened a new log
I20260927 06:59:10.050338 19218 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcdbb3b90016455eb3cb05520b6aa2dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38059 } }
I20260927 06:59:10.050686 19218 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:10.050804 19218 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcdbb3b90016455eb3cb05520b6aa2dc, State: Initialized, Role: FOLLOWER
I20260927 06:59:10.051409 19218 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcdbb3b90016455eb3cb05520b6aa2dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38059 } }
I20260927 06:59:10.051716 19218 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:10.051878 19218 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:10.052054 19218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:10.054201 19218 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcdbb3b90016455eb3cb05520b6aa2dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38059 } }
I20260927 06:59:10.054699 19218 leader_election.cc:302] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcdbb3b90016455eb3cb05520b6aa2dc; no voters:
I20260927 06:59:10.055240 19218 leader_election.cc:288] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:10.055373 19223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:10.056646 19223 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc [term 1 LEADER]: Becoming Leader. State: Replica: fcdbb3b90016455eb3cb05520b6aa2dc, State: Running, Role: LEADER
I20260927 06:59:10.057150 19223 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcdbb3b90016455eb3cb05520b6aa2dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38059 } }
I20260927 06:59:10.057294 19218 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:10.059697 19225 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcdbb3b90016455eb3cb05520b6aa2dc. Latest consensus state: current_term: 1 leader_uuid: "fcdbb3b90016455eb3cb05520b6aa2dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcdbb3b90016455eb3cb05520b6aa2dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38059 } } }
I20260927 06:59:10.060053 19225 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:10.060295 19224 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcdbb3b90016455eb3cb05520b6aa2dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcdbb3b90016455eb3cb05520b6aa2dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38059 } } }
I20260927 06:59:10.060626 19224 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:10.063330 19230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:10.067780 19230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:10.075508 19230 catalog_manager.cc:1347] Generated new cluster ID: 62d9dbfcd1644353a53b401db5e976e8
I20260927 06:59:10.075603 19230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:10.097550 19230 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:10.099466 19230 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:10.107347 19230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc: Generated new TSK 0
I20260927 06:59:10.108342 19230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:10.117077 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38059
--builtin_ntp_servers=127.0.97.20:40895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 06:59:10.437369 19242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:10.437690 19242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:10.437853 19242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:10.445060 19242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:10.445184 19242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:10.445379 19242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:59:10.454155 19242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:10.455437 19242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:10.457218 19242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:10.470471 19250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:10.470741 19247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:10.470557 19248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:10.471455 19242 server_base.cc:1034] running on GCE node
I20260927 06:59:10.472155 19242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:10.473080 19242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:10.474269 19242 hybrid_clock.cc:648] HybridClock initialized: now 1790492350474199 us; error 65 us; skew 500 ppm
I20260927 06:59:10.474617 19242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:59:10.476632 19242 webserver.cc:460] Webserver started at http://127.0.97.1:37839/ using document root <none> and password file <none>
I20260927 06:59:10.477221 19242 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:10.477300 19242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:10.477552 19242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:10.480278 19242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "3631d673fd054ac4952958a68d54b7bb"
format_stamp: "Formatted at 2026-09-27 06:59:10 on dist-test-slave-f892"
server_key: "d683b95943a9130b530bbaa85e0943cb"
server_key_iv: "078d7dd7a6a44dc5283af239c5e3fc4b"
server_key_version: "encryptionkey@0"
I20260927 06:59:10.480983 19242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3631d673fd054ac4952958a68d54b7bb"
format_stamp: "Formatted at 2026-09-27 06:59:10 on dist-test-slave-f892"
server_key: "d683b95943a9130b530bbaa85e0943cb"
server_key_iv: "078d7dd7a6a44dc5283af239c5e3fc4b"
server_key_version: "encryptionkey@0"
I20260927 06:59:10.485673 19242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260927 06:59:10.488790 19256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:10.490262 19242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:59:10.490429 19242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "3631d673fd054ac4952958a68d54b7bb"
format_stamp: "Formatted at 2026-09-27 06:59:10 on dist-test-slave-f892"
server_key: "d683b95943a9130b530bbaa85e0943cb"
server_key_iv: "078d7dd7a6a44dc5283af239c5e3fc4b"
server_key_version: "encryptionkey@0"
I20260927 06:59:10.490645 19242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:10.528469 19242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:10.529397 19242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:10.529696 19242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:10.530759 19242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:10.532331 19242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:10.532456 19242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:10.532591 19242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:10.532682 19242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:10.566591 19242 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42253
I20260927 06:59:10.566701 19368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42253 every 8 connection(s)
I20260927 06:59:10.567869 19242 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:59:10.573015 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 19242
I20260927 06:59:10.573177 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:59:10.573616 388 external_mini_cluster.cc:1428] Setting key fca993736983392179219082742369e1
I20260927 06:59:10.576592 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38059
--builtin_ntp_servers=127.0.97.20:40895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 06:59:10.581386 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.570422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59381 (local address 127.0.97.62:38059)
0927 06:59:10.570823 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:10.570828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:10.572068 (+ 1240us) server_negotiation.cc:407] Connection header received
0927 06:59:10.574089 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:10.574095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:10.574195 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:10.574303 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:10.575576 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.576640 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.578184 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.578888 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.579037 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.580364 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:10.580378 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:10.580386 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:10.580411 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:10.580465 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:10.580484 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:10.580575 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:10.580795 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 06:59:10.580910 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":207,"threads_started":1}
I20260927 06:59:10.582152 19372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.571038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38059 (local address 127.0.97.1:59381)
0927 06:59:10.571909 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:10.571954 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:10.573782 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:10.574544 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:10.574551 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:10.574669 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:10.575382 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.575406 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.576814 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.576819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.577980 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.577991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.579086 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.579090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.579279 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.580033 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:59:10.580036 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:10.580076 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:59:10.580156 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:10.580582 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:10.580590 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:10.580595 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 06:59:10.580735 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 06:59:10.581034 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":264,"threads_started":1}
I20260927 06:59:10.583642 19369 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38059
I20260927 06:59:10.583945 19369 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:10.584657 19369 heartbeater.cc:511] Master 127.0.97.62:38059 requested a full tablet report, sending...
I20260927 06:59:10.586560 19183 ts_manager.cc:194] Registered new tserver with Master: 3631d673fd054ac4952958a68d54b7bb (127.0.97.1:42253)
I20260927 06:59:10.587996 19183 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:59381
W20260927 06:59:10.900621 19373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:10.900887 19373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:10.901043 19373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:10.908237 19373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:10.908339 19373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:10.908536 19373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:59:10.917665 19373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:10.918983 19373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:10.920709 19373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:10.933096 19378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:10.933215 19379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:10.933430 19381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:10.934322 19373 server_base.cc:1034] running on GCE node
I20260927 06:59:10.934948 19373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:10.935817 19373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:10.937045 19373 hybrid_clock.cc:648] HybridClock initialized: now 1790492350936990 us; error 64 us; skew 500 ppm
I20260927 06:59:10.937337 19373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:59:10.939527 19373 webserver.cc:460] Webserver started at http://127.0.97.2:46559/ using document root <none> and password file <none>
I20260927 06:59:10.940165 19373 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:10.940294 19373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:10.940611 19373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:10.943312 19373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "e2252445dc574336904f097e7d48a7b7"
format_stamp: "Formatted at 2026-09-27 06:59:10 on dist-test-slave-f892"
server_key: "56bc747bb7497b616c9ac9a1ff3f8f69"
server_key_iv: "47d2af48f513f2da408f4e65559699aa"
server_key_version: "encryptionkey@0"
I20260927 06:59:10.944036 19373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e2252445dc574336904f097e7d48a7b7"
format_stamp: "Formatted at 2026-09-27 06:59:10 on dist-test-slave-f892"
server_key: "56bc747bb7497b616c9ac9a1ff3f8f69"
server_key_iv: "47d2af48f513f2da408f4e65559699aa"
server_key_version: "encryptionkey@0"
I20260927 06:59:10.948580 19373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 06:59:10.951683 19387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:10.952865 19373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 06:59:10.953037 19373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "e2252445dc574336904f097e7d48a7b7"
format_stamp: "Formatted at 2026-09-27 06:59:10 on dist-test-slave-f892"
server_key: "56bc747bb7497b616c9ac9a1ff3f8f69"
server_key_iv: "47d2af48f513f2da408f4e65559699aa"
server_key_version: "encryptionkey@0"
I20260927 06:59:10.953251 19373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:10.970440 19373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:10.971185 19373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:10.971447 19373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:10.972337 19373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:10.973721 19373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:10.973834 19373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:10.973954 19373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:10.974067 19373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:11.006510 19373 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46007
I20260927 06:59:11.006597 19499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46007 every 8 connection(s)
I20260927 06:59:11.007815 19373 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:59:11.010296 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 19373
I20260927 06:59:11.010476 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:59:11.010922 388 external_mini_cluster.cc:1428] Setting key 7c965e519d63514b46b0e38bd515a543
I20260927 06:59:11.013862 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38059
--builtin_ntp_servers=127.0.97.20:40895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 06:59:11.022235 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.010193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51409 (local address 127.0.97.62:38059)
0927 06:59:11.010426 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:11.010432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:11.011918 (+ 1486us) server_negotiation.cc:407] Connection header received
0927 06:59:11.014242 (+ 2324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:11.014246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:11.014311 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:11.014397 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:11.015762 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.016449 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.018841 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.019497 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.019653 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:11.021332 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:11.021339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:11.021344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:11.021367 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:11.021406 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:11.021420 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:11.021527 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:11.021770 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 06:59:11.021879 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 06:59:11.023645 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.010741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38059 (local address 127.0.97.2:51409)
0927 06:59:11.011754 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:11.011801 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:11.013923 (+ 2122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:11.014652 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:11.014667 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:11.014786 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:11.015583 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:11.015608 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.016600 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.016609 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:11.018679 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:11.018696 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.019765 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.019774 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:11.020114 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:11.020968 (+ 854us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:59:11.020974 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:11.021035 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:59:11.021139 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:11.021533 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:11.021546 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:11.021557 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 06:59:11.021739 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 06:59:11.022204 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":331,"threads_started":1}
I20260927 06:59:11.025171 19500 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38059
I20260927 06:59:11.025506 19500 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:11.026367 19500 heartbeater.cc:511] Master 127.0.97.62:38059 requested a full tablet report, sending...
I20260927 06:59:11.027788 19183 ts_manager.cc:194] Registered new tserver with Master: e2252445dc574336904f097e7d48a7b7 (127.0.97.2:46007)
I20260927 06:59:11.028630 19183 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:51409
W20260927 06:59:11.355743 19503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:11.356056 19503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:11.356181 19503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:11.363371 19503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:11.363461 19503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:11.363603 19503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:59:11.372728 19503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:11.373936 19503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:11.375597 19503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:11.387637 19509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:11.387620 19508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:11.387943 19511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:11.388454 19503 server_base.cc:1034] running on GCE node
I20260927 06:59:11.389017 19503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:11.389890 19503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:11.391165 19503 hybrid_clock.cc:648] HybridClock initialized: now 1790492351391138 us; error 76 us; skew 500 ppm
I20260927 06:59:11.391510 19503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:59:11.393620 19503 webserver.cc:460] Webserver started at http://127.0.97.3:44825/ using document root <none> and password file <none>
I20260927 06:59:11.394419 19503 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:11.394539 19503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:11.394858 19503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:11.397615 19503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "f633da9780b140058e3f894f3e0a1cc9"
format_stamp: "Formatted at 2026-09-27 06:59:11 on dist-test-slave-f892"
server_key: "5848e6986217b256ada3898a6d72123e"
server_key_iv: "02b3d982b6a148c65bde5f88883f24c3"
server_key_version: "encryptionkey@0"
I20260927 06:59:11.398371 19503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f633da9780b140058e3f894f3e0a1cc9"
format_stamp: "Formatted at 2026-09-27 06:59:11 on dist-test-slave-f892"
server_key: "5848e6986217b256ada3898a6d72123e"
server_key_iv: "02b3d982b6a148c65bde5f88883f24c3"
server_key_version: "encryptionkey@0"
I20260927 06:59:11.403062 19503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 06:59:11.406283 19517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:11.407526 19503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 06:59:11.407644 19503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "f633da9780b140058e3f894f3e0a1cc9"
format_stamp: "Formatted at 2026-09-27 06:59:11 on dist-test-slave-f892"
server_key: "5848e6986217b256ada3898a6d72123e"
server_key_iv: "02b3d982b6a148c65bde5f88883f24c3"
server_key_version: "encryptionkey@0"
I20260927 06:59:11.407809 19503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:11.433359 19503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:11.434229 19503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:11.434470 19503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:11.435400 19503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:11.436800 19503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:11.436885 19503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:11.436964 19503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:11.437021 19503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:11.469451 19503 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35107
I20260927 06:59:11.469542 19629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35107 every 8 connection(s)
I20260927 06:59:11.470747 19503 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:59:11.473702 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 19503
I20260927 06:59:11.473881 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:59:11.474561 388 external_mini_cluster.cc:1428] Setting key 7262ccb2483d987c8789a3a047583814
I20260927 06:59:11.483413 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.473245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50467 (local address 127.0.97.62:38059)
0927 06:59:11.473454 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:11.473460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:11.475092 (+ 1632us) server_negotiation.cc:407] Connection header received
0927 06:59:11.477250 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:11.477256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:11.477336 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:11.477451 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:11.478506 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.479272 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.480747 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.481171 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.481305 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:11.482612 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:11.482619 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:11.482623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:11.482650 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:11.482680 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:11.482691 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:11.482789 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:11.483004 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 06:59:11.483096 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 06:59:11.484257 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.473757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38059 (local address 127.0.97.3:50467)
0927 06:59:11.474915 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:11.474967 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:11.476959 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:11.477676 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:11.477684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:11.477766 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:11.478355 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:11.478371 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.479418 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.479423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:11.480592 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:11.480603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.481390 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.481396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:11.481610 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:11.482320 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:59:11.482322 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:11.482356 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:59:11.482425 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:11.482788 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:11.482799 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:11.482809 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 06:59:11.482976 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 06:59:11.483321 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":374,"threads_started":1}
I20260927 06:59:11.485363 19630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38059
I20260927 06:59:11.485630 19630 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:11.486346 19630 heartbeater.cc:511] Master 127.0.97.62:38059 requested a full tablet report, sending...
I20260927 06:59:11.487687 19183 ts_manager.cc:194] Registered new tserver with Master: f633da9780b140058e3f894f3e0a1cc9 (127.0.97.3:35107)
I20260927 06:59:11.488627 19183 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:50467
I20260927 06:59:11.489464 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:59:11.498844 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.493501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.0.97.62:38059)
0927 06:59:11.493677 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:11.493681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:11.494056 (+ 375us) server_negotiation.cc:407] Connection header received
0927 06:59:11.494284 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:11.494291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:11.494377 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:11.494494 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:11.494896 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.495605 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.496827 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.497207 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.497295 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:11.498305 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:11.498310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:11.498313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:11.498328 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:11.498355 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:11.498366 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:11.498478 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:11.498605 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 06:59:11.498644 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 06:59:11.502703 19182 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:59:11.509387 19182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:59:11.531659 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.521605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47786 (local address 127.0.97.2:46007)
0927 06:59:11.523118 (+ 1513us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:11.523137 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:11.523165 (+ 28us) server_negotiation.cc:407] Connection header received
0927 06:59:11.523264 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:11.523273 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:11.523515 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:11.525012 (+ 1497us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:11.525204 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.526763 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.529353 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.530548 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.530695 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:11.530926 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:11.531197 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 06:59:11.531346 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1341,"thread_start_us":2035,"threads_started":1}
I20260927 06:59:11.531700 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.520900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46007 (local address 127.0.0.1:47786)
0927 06:59:11.522090 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:11.522145 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:11.522343 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:11.523738 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:11.523747 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:11.523779 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:11.524401 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:11.524425 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.526935 (+ 2510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.526940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:11.529106 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:11.529139 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.530735 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.530741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:11.530964 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:11.531032 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 06:59:11.531158 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 06:59:11.531324 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":98,"thread_start_us":1262,"threads_started":1}
I20260927 06:59:11.532635 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.521141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42253 (local address 127.0.0.1:49454)
0927 06:59:11.523077 (+ 1936us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:11.523102 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:11.523224 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:11.524305 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:11.524310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:11.524347 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:11.525237 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:11.525252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.527033 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.527042 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:11.529756 (+ 2714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:11.529770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.531599 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.531606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:11.532059 (+ 453us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:11.532093 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 06:59:11.532182 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 06:59:11.532296 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1763,"mutex_wait_us":6,"spinlock_wait_cycles":883456,"thread_start_us":1568,"threads_started":1}
I20260927 06:59:11.533061 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.521750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49454 (local address 127.0.97.1:42253)
0927 06:59:11.523702 (+ 1952us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:11.523709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:11.523737 (+ 28us) server_negotiation.cc:407] Connection header received
0927 06:59:11.523834 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:11.523844 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:11.524099 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:11.524599 (+ 500us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:11.525417 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.526868 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.529955 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.531387 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.532315 (+ 928us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:11.532522 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:11.532583 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 06:59:11.532735 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1813,"thread_start_us":1707,"threads_started":1}
I20260927 06:59:11.534196 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.523152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35107 (local address 127.0.0.1:37790)
0927 06:59:11.524283 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:11.524304 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:11.524823 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:11.526631 (+ 1808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:11.526636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:11.526659 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:11.527041 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:11.527057 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.529044 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.529052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:11.530926 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:11.530939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.533497 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.533502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:11.533612 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:11.533634 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 06:59:11.533742 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 06:59:11.533856 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"mutex_wait_us":1,"thread_start_us":846,"threads_started":1}
I20260927 06:59:11.534334 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.523677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.0.97.3:35107)
0927 06:59:11.526227 (+ 2550us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:11.526232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:11.526256 (+ 24us) server_negotiation.cc:407] Connection header received
0927 06:59:11.526325 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:11.526330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:11.526459 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:11.526632 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:11.527267 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.528873 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.531129 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.533188 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.533310 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:11.533565 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:11.533792 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 06:59:11.533940 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2415,"thread_start_us":2265,"threads_started":1}
I20260927 06:59:11.535753 19435 tablet_service.cc:1467] Processing CreateTablet for tablet 9cbc304bf6604364a4a1da08d0f8fce0 (DEFAULT_TABLE table=test-table [id=7ddf7c856916462aae03a4ea14768c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:11.536902 19435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cbc304bf6604364a4a1da08d0f8fce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:11.537019 19304 tablet_service.cc:1467] Processing CreateTablet for tablet 9cbc304bf6604364a4a1da08d0f8fce0 (DEFAULT_TABLE table=test-table [id=7ddf7c856916462aae03a4ea14768c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:11.537768 19565 tablet_service.cc:1467] Processing CreateTablet for tablet 9cbc304bf6604364a4a1da08d0f8fce0 (DEFAULT_TABLE table=test-table [id=7ddf7c856916462aae03a4ea14768c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:11.538393 19304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cbc304bf6604364a4a1da08d0f8fce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:11.539088 19565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cbc304bf6604364a4a1da08d0f8fce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:11.548600 19648 tablet_bootstrap.cc:492] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7: Bootstrap starting.
I20260927 06:59:11.552218 19648 tablet_bootstrap.cc:654] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:11.553254 19649 tablet_bootstrap.cc:492] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb: Bootstrap starting.
I20260927 06:59:11.553488 19648 log.cc:824] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:11.554078 19650 tablet_bootstrap.cc:492] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9: Bootstrap starting.
I20260927 06:59:11.557030 19649 tablet_bootstrap.cc:654] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:11.558235 19649 log.cc:824] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:11.558424 19650 tablet_bootstrap.cc:654] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:11.559643 19650 log.cc:824] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:11.560835 19648 tablet_bootstrap.cc:492] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7: No bootstrap required, opened a new log
I20260927 06:59:11.561178 19648 ts_tablet_manager.cc:1397] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260927 06:59:11.561842 19649 tablet_bootstrap.cc:492] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb: No bootstrap required, opened a new log
I20260927 06:59:11.562106 19649 ts_tablet_manager.cc:1397] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 06:59:11.568032 19650 tablet_bootstrap.cc:492] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9: No bootstrap required, opened a new log
I20260927 06:59:11.568289 19650 ts_tablet_manager.cc:1397] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260927 06:59:11.572157 19649 raft_consensus.cc:348] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2252445dc574336904f097e7d48a7b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46007 } } peers { permanent_uuid: "3631d673fd054ac4952958a68d54b7bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42253 } } peers { permanent_uuid: "f633da9780b140058e3f894f3e0a1cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35107 } }
I20260927 06:59:11.572582 19649 raft_consensus.cc:374] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:11.572711 19649 raft_consensus.cc:729] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3631d673fd054ac4952958a68d54b7bb, State: Initialized, Role: FOLLOWER
I20260927 06:59:11.572731 19648 raft_consensus.cc:348] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2252445dc574336904f097e7d48a7b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46007 } } peers { permanent_uuid: "3631d673fd054ac4952958a68d54b7bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42253 } } peers { permanent_uuid: "f633da9780b140058e3f894f3e0a1cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35107 } }
I20260927 06:59:11.573297 19648 raft_consensus.cc:374] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:11.573431 19648 raft_consensus.cc:729] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2252445dc574336904f097e7d48a7b7, State: Initialized, Role: FOLLOWER
I20260927 06:59:11.573377 19649 consensus_queue.cc:260] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2252445dc574336904f097e7d48a7b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46007 } } peers { permanent_uuid: "3631d673fd054ac4952958a68d54b7bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42253 } } peers { permanent_uuid: "f633da9780b140058e3f894f3e0a1cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35107 } }
I20260927 06:59:11.574170 19648 consensus_queue.cc:260] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2252445dc574336904f097e7d48a7b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46007 } } peers { permanent_uuid: "3631d673fd054ac4952958a68d54b7bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42253 } } peers { permanent_uuid: "f633da9780b140058e3f894f3e0a1cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35107 } }
I20260927 06:59:11.581666 19650 raft_consensus.cc:348] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2252445dc574336904f097e7d48a7b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46007 } } peers { permanent_uuid: "3631d673fd054ac4952958a68d54b7bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42253 } } peers { permanent_uuid: "f633da9780b140058e3f894f3e0a1cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35107 } }
I20260927 06:59:11.582309 19650 raft_consensus.cc:374] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:11.582373 19648 ts_tablet_manager.cc:1428] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260927 06:59:11.582474 19650 raft_consensus.cc:729] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f633da9780b140058e3f894f3e0a1cc9, State: Initialized, Role: FOLLOWER
I20260927 06:59:11.582821 19500 heartbeater.cc:503] Master 127.0.97.62:38059 was elected leader, sending a full tablet report...
I20260927 06:59:11.585248 19650 consensus_queue.cc:260] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2252445dc574336904f097e7d48a7b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46007 } } peers { permanent_uuid: "3631d673fd054ac4952958a68d54b7bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42253 } } peers { permanent_uuid: "f633da9780b140058e3f894f3e0a1cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35107 } }
I20260927 06:59:11.585940 19649 ts_tablet_manager.cc:1428] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb: Time spent starting tablet: real 0.024s user 0.015s sys 0.005s
I20260927 06:59:11.585951 19369 heartbeater.cc:503] Master 127.0.97.62:38059 was elected leader, sending a full tablet report...
I20260927 06:59:11.588105 19630 heartbeater.cc:503] Master 127.0.97.62:38059 was elected leader, sending a full tablet report...
I20260927 06:59:11.588111 19650 ts_tablet_manager.cc:1428] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9: Time spent starting tablet: real 0.020s user 0.013s sys 0.004s
I20260927 06:59:11.681955 19654 raft_consensus.cc:482] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:59:11.682391 19654 raft_consensus.cc:504] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2252445dc574336904f097e7d48a7b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46007 } } peers { permanent_uuid: "3631d673fd054ac4952958a68d54b7bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42253 } } peers { permanent_uuid: "f633da9780b140058e3f894f3e0a1cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35107 } }
I20260927 06:59:11.684136 19654 leader_election.cc:288] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2252445dc574336904f097e7d48a7b7 (127.0.97.2:46007), f633da9780b140058e3f894f3e0a1cc9 (127.0.97.3:35107)
I20260927 06:59:11.690698 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.684425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34985 (local address 127.0.97.2:46007)
0927 06:59:11.684664 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:11.684673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:11.685515 (+ 842us) server_negotiation.cc:407] Connection header received
0927 06:59:11.685744 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:11.685754 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:11.685834 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:11.685929 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:11.686526 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.687216 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.689034 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.689773 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.689911 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:11.690007 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:11.690314 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 06:59:11.690431 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 06:59:11.690913 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.684664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52467 (local address 127.0.97.3:35107)
0927 06:59:11.684910 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:11.684919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:11.685703 (+ 784us) server_negotiation.cc:407] Connection header received
0927 06:59:11.685908 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:11.685912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:11.685967 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:11.686106 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:11.686712 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.687425 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.689219 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.689986 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.690211 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:11.690306 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:11.690560 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 06:59:11.690699 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 06:59:11.691083 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.684490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35107 (local address 127.0.97.1:52467)
0927 06:59:11.685592 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:11.685606 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:11.685748 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:11.686125 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:11.686129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:11.686164 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:11.686559 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:11.686568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.687578 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.687584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:11.689039 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:11.689051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.690239 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.690245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:11.690409 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:11.690431 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 06:59:11.690534 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 06:59:11.690656 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"mutex_wait_us":6,"thread_start_us":763,"threads_started":1}
I20260927 06:59:11.691145 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.684291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46007 (local address 127.0.97.1:34985)
0927 06:59:11.685401 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:11.685417 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:11.685594 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:11.685943 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:11.685947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:11.685969 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:11.686377 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:11.686386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.687354 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.687361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:11.688842 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:11.688853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.689963 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.689969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:11.690154 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:11.690176 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 06:59:11.690289 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 06:59:11.690414 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"mutex_wait_us":20,"spinlock_wait_cycles":16768,"thread_start_us":656,"threads_started":1}
I20260927 06:59:11.692168 19455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cbc304bf6604364a4a1da08d0f8fce0" candidate_uuid: "3631d673fd054ac4952958a68d54b7bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2252445dc574336904f097e7d48a7b7" is_pre_election: true
I20260927 06:59:11.692163 19585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cbc304bf6604364a4a1da08d0f8fce0" candidate_uuid: "3631d673fd054ac4952958a68d54b7bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f633da9780b140058e3f894f3e0a1cc9" is_pre_election: true
I20260927 06:59:11.692621 19585 raft_consensus.cc:2454] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631d673fd054ac4952958a68d54b7bb in term 0.
I20260927 06:59:11.692632 19455 raft_consensus.cc:2454] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631d673fd054ac4952958a68d54b7bb in term 0.
I20260927 06:59:11.693264 19260 leader_election.cc:302] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3631d673fd054ac4952958a68d54b7bb, e2252445dc574336904f097e7d48a7b7, f633da9780b140058e3f894f3e0a1cc9; no voters:
I20260927 06:59:11.693647 19654 raft_consensus.cc:2781] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:59:11.693809 19654 raft_consensus.cc:482] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:59:11.693919 19654 raft_consensus.cc:3037] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:11.696161 19654 raft_consensus.cc:504] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2252445dc574336904f097e7d48a7b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46007 } } peers { permanent_uuid: "3631d673fd054ac4952958a68d54b7bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42253 } } peers { permanent_uuid: "f633da9780b140058e3f894f3e0a1cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35107 } }
I20260927 06:59:11.696728 19654 leader_election.cc:288] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [CANDIDATE]: Term 1 election: Requested vote from peers e2252445dc574336904f097e7d48a7b7 (127.0.97.2:46007), f633da9780b140058e3f894f3e0a1cc9 (127.0.97.3:35107)
I20260927 06:59:11.697046 19455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cbc304bf6604364a4a1da08d0f8fce0" candidate_uuid: "3631d673fd054ac4952958a68d54b7bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2252445dc574336904f097e7d48a7b7"
I20260927 06:59:11.697150 19585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cbc304bf6604364a4a1da08d0f8fce0" candidate_uuid: "3631d673fd054ac4952958a68d54b7bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f633da9780b140058e3f894f3e0a1cc9"
I20260927 06:59:11.697265 19455 raft_consensus.cc:3037] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:11.697386 19585 raft_consensus.cc:3037] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:11.699436 19455 raft_consensus.cc:2454] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631d673fd054ac4952958a68d54b7bb in term 1.
I20260927 06:59:11.699572 19585 raft_consensus.cc:2454] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631d673fd054ac4952958a68d54b7bb in term 1.
I20260927 06:59:11.699802 19260 leader_election.cc:302] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3631d673fd054ac4952958a68d54b7bb, e2252445dc574336904f097e7d48a7b7; no voters:
I20260927 06:59:11.700094 19654 raft_consensus.cc:2781] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:11.700527 19654 raft_consensus.cc:686] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [term 1 LEADER]: Becoming Leader. State: Replica: 3631d673fd054ac4952958a68d54b7bb, State: Running, Role: LEADER
I20260927 06:59:11.701233 19654 consensus_queue.cc:237] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2252445dc574336904f097e7d48a7b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46007 } } peers { permanent_uuid: "3631d673fd054ac4952958a68d54b7bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42253 } } peers { permanent_uuid: "f633da9780b140058e3f894f3e0a1cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35107 } }
I20260927 06:59:11.705543 19182 catalog_manager.cc:5495] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 3631d673fd054ac4952958a68d54b7bb (127.0.97.1). New cstate: current_term: 1 leader_uuid: "3631d673fd054ac4952958a68d54b7bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2252445dc574336904f097e7d48a7b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631d673fd054ac4952958a68d54b7bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f633da9780b140058e3f894f3e0a1cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35107 } health_report { overall_health: UNKNOWN } } }
W20260927 06:59:11.722304 19631 tablet.cc:2369] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:59:11.723637 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.717385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49468 (local address 127.0.97.1:42253)
0927 06:59:11.717598 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:11.717604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:11.717706 (+ 102us) server_negotiation.cc:407] Connection header received
0927 06:59:11.717953 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:11.717957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:11.718110 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:11.718265 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:11.718634 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.719329 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.721408 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.721845 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.721982 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:11.722597 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:11.723057 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:11.723244 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:11.723362 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 06:59:11.723406 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 06:59:11.731530 19455 raft_consensus.cc:1261] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7 [term 1 FOLLOWER]: Refusing update from remote peer 3631d673fd054ac4952958a68d54b7bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:11.731539 19585 raft_consensus.cc:1261] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9 [term 1 FOLLOWER]: Refusing update from remote peer 3631d673fd054ac4952958a68d54b7bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:11.732546 19659 consensus_queue.cc:1035] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f633da9780b140058e3f894f3e0a1cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:11.732923 19654 consensus_queue.cc:1035] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2252445dc574336904f097e7d48a7b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:11.759141 19182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:11.759413 19182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:11.763157 19182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcdbb3b90016455eb3cb05520b6aa2dc: Sending DeleteTablet for 3 replicas of tablet 9cbc304bf6604364a4a1da08d0f8fce0
I20260927 06:59:11.764277 19435 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cbc304bf6604364a4a1da08d0f8fce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:11 UTC) from {username='slave'} at 127.0.0.1:47786
I20260927 06:59:11.764575 19565 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cbc304bf6604364a4a1da08d0f8fce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:11 UTC) from {username='slave'} at 127.0.0.1:37790
I20260927 06:59:11.764657 19304 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cbc304bf6604364a4a1da08d0f8fce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:11 UTC) from {username='slave'} at 127.0.0.1:49454
I20260927 06:59:11.766341 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 19242
I20260927 06:59:11.774557 19672 tablet_replica.cc:330] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9: stopping tablet replica
I20260927 06:59:11.774705 19671 tablet_replica.cc:330] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7: stopping tablet replica
I20260927 06:59:11.775146 19671 raft_consensus.cc:2229] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:11.775143 19672 raft_consensus.cc:2229] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:11.775637 19671 raft_consensus.cc:2258] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:11.775637 19672 raft_consensus.cc:2258] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:11.777637 19671 ts_tablet_manager.cc:1905] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:11.777637 19672 ts_tablet_manager.cc:1905] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:11.778469 19673 tablet_replica.cc:330] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb: stopping tablet replica
I20260927 06:59:11.778857 19673 raft_consensus.cc:2229] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:59:11.779520 19673 raft_consensus.cc:2258] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:11.785951 19671 ts_tablet_manager.cc:1918] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:11.786072 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.767606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49474 (local address 127.0.97.1:42253)
0927 06:59:11.767811 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:11.767817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:11.773552 (+ 5735us) server_negotiation.cc:407] Connection header received
0927 06:59:11.773769 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:11.773774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:11.773878 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:11.773981 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:11.778783 (+ 4802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.780026 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.781933 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.782668 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.782789 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:11.784232 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:11.784243 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:11.784261 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:11.785101 (+ 840us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:11.785169 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:11.785193 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:11.785285 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:11.785536 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 06:59:11.785658 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 06:59:11.786841 19672 ts_tablet_manager.cc:1918] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:11.787141 19672 log.cc:1196] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/9cbc304bf6604364a4a1da08d0f8fce0
I20260927 06:59:11.787482 19364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:11.788022 19672 ts_tablet_manager.cc:1939] T 9cbc304bf6604364a4a1da08d0f8fce0 P f633da9780b140058e3f894f3e0a1cc9: Deleting consensus metadata
I20260927 06:59:11.788401 19671 log.cc:1196] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/9cbc304bf6604364a4a1da08d0f8fce0
I20260927 06:59:11.788862 19671 ts_tablet_manager.cc:1939] T 9cbc304bf6604364a4a1da08d0f8fce0 P e2252445dc574336904f097e7d48a7b7: Deleting consensus metadata
I20260927 06:59:11.790333 19167 catalog_manager.cc:4841] TS f633da9780b140058e3f894f3e0a1cc9 (127.0.97.3:35107): tablet 9cbc304bf6604364a4a1da08d0f8fce0 (table test-table [id=7ddf7c856916462aae03a4ea14768c7b]) successfully deleted
I20260927 06:59:11.790675 19170 catalog_manager.cc:4841] TS e2252445dc574336904f097e7d48a7b7 (127.0.97.2:46007): tablet 9cbc304bf6604364a4a1da08d0f8fce0 (table test-table [id=7ddf7c856916462aae03a4ea14768c7b]) successfully deleted
I20260927 06:59:11.788270 19673 ts_tablet_manager.cc:1905] T 9cbc304bf6604364a4a1da08d0f8fce0 P 3631d673fd054ac4952958a68d54b7bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:11.885673 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 19242
I20260927 06:59:11.903704 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 19373
W20260927 06:59:11.903750 19167 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42253 (error 108)
I20260927 06:59:11.905623 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.905189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42253 (local address 127.0.0.1:49478)
0927 06:59:11.905350 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:11.905462 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 06:59:11.906214 19167 catalog_manager.cc:4584] TS 3631d673fd054ac4952958a68d54b7bb (127.0.97.1:42253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9cbc304bf6604364a4a1da08d0f8fce0: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42253: connect: Connection refused (error 111)
I20260927 06:59:11.910475 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.904510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47788 (local address 127.0.97.2:46007)
0927 06:59:11.904707 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:11.904714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:11.904787 (+ 73us) server_negotiation.cc:407] Connection header received
0927 06:59:11.905002 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:11.905008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:11.905116 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:11.905230 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:11.905691 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.906586 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.907753 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:11.908128 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:11.908217 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:11.909099 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:11.909108 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:11.909120 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:11.909700 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:11.909744 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:11.909763 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:11.909855 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:11.909975 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 06:59:11.910113 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 06:59:11.911198 19495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:11.959726 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.959208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42253 (local address 127.0.0.1:49480)
0927 06:59:11.959425 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:11.959528 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 06:59:11.960758 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:11.960367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42253 (local address 127.0.0.1:49496)
0927 06:59:11.960517 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:11.960620 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 06:59:12.000331 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 19373
I20260927 06:59:12.017333 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 19503
I20260927 06:59:12.021278 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:12.020815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42253 (local address 127.0.0.1:49506)
0927 06:59:12.021042 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:12.021119 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 06:59:12.022595 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:12.022218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42253 (local address 127.0.0.1:49510)
0927 06:59:12.022373 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:12.022468 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 06:59:12.025556 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:12.018133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.0.97.3:35107)
0927 06:59:12.018295 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:12.018299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:12.018317 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:59:12.018374 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:12.018378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:12.018465 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:12.018556 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:12.019075 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.019902 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.021797 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.022583 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.022719 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:12.023982 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:12.023996 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:12.024017 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:12.024768 (+ 751us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:12.024836 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:12.024866 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:12.024978 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:12.025122 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 06:59:12.025241 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 06:59:12.026365 19625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:12.103544 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:12.103008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42253 (local address 127.0.0.1:49522)
0927 06:59:12.103250 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:12.103352 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 06:59:12.104679 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:12.104274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42253 (local address 127.0.0.1:49524)
0927 06:59:12.104465 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:12.104555 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260927 06:59:12.116325 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 19503
I20260927 06:59:12.133455 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 19152
I20260927 06:59:12.134593 19213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:12.243429 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 19152
2026-09-27T06:59:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2719 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:12 dist-test-slave-f892 krb5kdc[19682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:12 dist-test-slave-f892 krb5kdc[19682](info): set up 2 sockets
Sep 27 06:59:12 dist-test-slave-f892 krb5kdc[19682](info): commencing operation
krb5kdc: starting...
W20260927 06:59:13.350106 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:13 dist-test-slave-f892 krb5kdc[19682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:59:13.586905 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37001
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:13.901381 19695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:13.901700 19695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:13.901871 19695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:13.909008 19695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:13.909164 19695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:13.909279 19695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:13.909385 19695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:13.909452 19695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:13.919338 19695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:13.920738 19695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:13.922076 19695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:13.933801 19700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:13.933842 19701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:13.933750 19703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:13.934965 19695 server_base.cc:1034] running on GCE node
I20260927 06:59:13.935747 19695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:13.937124 19695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:13.938362 19695 hybrid_clock.cc:648] HybridClock initialized: now 1790492353938319 us; error 54 us; skew 500 ppm
I20260927 06:59:13.938712 19695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:13 dist-test-slave-f892 krb5kdc[19682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:13.945209 19695 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:59:13.946758 19695 webserver.cc:460] Webserver started at http://127.0.97.62:41645/ using document root <none> and password file <none>
I20260927 06:59:13.947402 19695 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:13.947500 19695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:13.947753 19695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:13.950588 19695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "3956ec59bd8242eab20cdb5f728b058c"
format_stamp: "Formatted at 2026-09-27 06:59:13 on dist-test-slave-f892"
server_key: "674479ef6b8c3e37499c86ae4cfd0667"
server_key_iv: "b321cff0223277f1e586206189556892"
server_key_version: "encryptionkey@0"
I20260927 06:59:13.951265 19695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "3956ec59bd8242eab20cdb5f728b058c"
format_stamp: "Formatted at 2026-09-27 06:59:13 on dist-test-slave-f892"
server_key: "674479ef6b8c3e37499c86ae4cfd0667"
server_key_iv: "b321cff0223277f1e586206189556892"
server_key_version: "encryptionkey@0"
I20260927 06:59:13.955901 19695 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 06:59:13.959102 19710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:13.960307 19695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:59:13.960439 19695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "3956ec59bd8242eab20cdb5f728b058c"
format_stamp: "Formatted at 2026-09-27 06:59:13 on dist-test-slave-f892"
server_key: "674479ef6b8c3e37499c86ae4cfd0667"
server_key_iv: "b321cff0223277f1e586206189556892"
server_key_version: "encryptionkey@0"
I20260927 06:59:13.960598 19695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:13.976181 19695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:13.980196 19695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:13.980409 19695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:13.998478 19695 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37001
I20260927 06:59:13.998521 19761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37001 every 8 connection(s)
I20260927 06:59:13.999790 19695 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:14.003223 19762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:14.008096 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 19695
I20260927 06:59:14.008296 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:14.008736 388 external_mini_cluster.cc:1428] Setting key 4d6e53c541a6141d63b6ac8466d72c4d
I20260927 06:59:14.013405 19762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c: Bootstrap starting.
I20260927 06:59:14.017164 19762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:14.018427 19762 log.cc:824] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:14.021654 19762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c: No bootstrap required, opened a new log
Sep 27 06:59:14 dist-test-slave-f892 krb5kdc[19682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:59:14.029165 19762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3956ec59bd8242eab20cdb5f728b058c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37001 } }
I20260927 06:59:14.029542 19762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:14.029673 19762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3956ec59bd8242eab20cdb5f728b058c, State: Initialized, Role: FOLLOWER
I20260927 06:59:14.030344 19762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3956ec59bd8242eab20cdb5f728b058c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37001 } }
I20260927 06:59:14.030598 19762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:14.030743 19762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:14.030956 19762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:14.032279 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:14.010453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48236 (local address 127.0.97.62:37001)
0927 06:59:14.011280 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:14.011291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:14.011336 (+ 45us) server_negotiation.cc:407] Connection header received
0927 06:59:14.013046 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:14.013070 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:14.013429 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:14.013792 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:14.015681 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:14.016945 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:14.018824 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:14.020005 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:14.020202 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:14.024608 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:14.024616 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:14.024630 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:14.024672 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:14.027949 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:14.028669 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:14.028676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:14.028681 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:14.028844 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:14.029267 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:14.029276 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:14.029278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:14.029943 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:14.030311 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:14.030802 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 06:59:14.031246 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":420,"threads_started":1}
I20260927 06:59:14.033712 19762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3956ec59bd8242eab20cdb5f728b058c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37001 } }
I20260927 06:59:14.034286 19762 leader_election.cc:302] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3956ec59bd8242eab20cdb5f728b058c; no voters:
I20260927 06:59:14.034883 19762 leader_election.cc:288] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:14.035087 19767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:14.035872 19767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c [term 1 LEADER]: Becoming Leader. State: Replica: 3956ec59bd8242eab20cdb5f728b058c, State: Running, Role: LEADER
I20260927 06:59:14.036428 19767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3956ec59bd8242eab20cdb5f728b058c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37001 } }
I20260927 06:59:14.037086 19762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:14.039264 19768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3956ec59bd8242eab20cdb5f728b058c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3956ec59bd8242eab20cdb5f728b058c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37001 } } }
I20260927 06:59:14.039729 19768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:14.043931 19769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3956ec59bd8242eab20cdb5f728b058c. Latest consensus state: current_term: 1 leader_uuid: "3956ec59bd8242eab20cdb5f728b058c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3956ec59bd8242eab20cdb5f728b058c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37001 } } }
I20260927 06:59:14.044361 19769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c [sys.catalog]: This master's current role is: LEADER
W20260927 06:59:14.047734 19783 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 06:59:14.047878 19783 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 06:59:14.048039 19776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:14.050184 19776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:14.059182 19776 catalog_manager.cc:1347] Generated new cluster ID: 20d66ddf1ff644ed81e2e7f1b3c49e55
I20260927 06:59:14.059291 19776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:14.069970 19776 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:14.071676 19776 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:14.083232 19776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c: Generated new TSK 0
I20260927 06:59:14.084144 19776 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 06:59:14.137583 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37001
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:14.462592 19788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:14.462841 19788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:14.463019 19788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:14.470444 19788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:14.470561 19788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:14.470747 19788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:59:14.480096 19788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:14.481267 19788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:14.482542 19788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:14.494982 19796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:14.495030 19793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:14.494930 19794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:14.496016 19788 server_base.cc:1034] running on GCE node
I20260927 06:59:14.496614 19788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:14.497460 19788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:14.498690 19788 hybrid_clock.cc:648] HybridClock initialized: now 1790492354498646 us; error 37 us; skew 500 ppm
I20260927 06:59:14.499004 19788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:14 dist-test-slave-f892 krb5kdc[19682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:14.501921 19788 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 06:59:14.503355 19788 webserver.cc:460] Webserver started at http://127.0.97.1:35057/ using document root <none> and password file <none>
I20260927 06:59:14.504011 19788 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:14.504156 19788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:14.504459 19788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:14.507148 19788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "8b2bd73cb8c24a44a4cd033787f44774"
format_stamp: "Formatted at 2026-09-27 06:59:14 on dist-test-slave-f892"
server_key: "31ffc6274c732f236655d98df44dc790"
server_key_iv: "82826a03c3928c24377940eefc34bf0b"
server_key_version: "encryptionkey@0"
I20260927 06:59:14.507939 19788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8b2bd73cb8c24a44a4cd033787f44774"
format_stamp: "Formatted at 2026-09-27 06:59:14 on dist-test-slave-f892"
server_key: "31ffc6274c732f236655d98df44dc790"
server_key_iv: "82826a03c3928c24377940eefc34bf0b"
server_key_version: "encryptionkey@0"
I20260927 06:59:14.512722 19788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 06:59:14.516059 19803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:14.517278 19788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 06:59:14.517469 19788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "8b2bd73cb8c24a44a4cd033787f44774"
format_stamp: "Formatted at 2026-09-27 06:59:14 on dist-test-slave-f892"
server_key: "31ffc6274c732f236655d98df44dc790"
server_key_iv: "82826a03c3928c24377940eefc34bf0b"
server_key_version: "encryptionkey@0"
I20260927 06:59:14.517686 19788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:14.545781 19788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:14.549675 19788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:14.549969 19788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:14.550952 19788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:14.552234 19788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:14.552346 19788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:14.552498 19788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:14.552585 19788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:14.585276 19788 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38229
I20260927 06:59:14.585397 19915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38229 every 8 connection(s)
I20260927 06:59:14.586635 19788 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:59:14.590389 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 19788
I20260927 06:59:14.590551 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:59:14.591069 388 external_mini_cluster.cc:1428] Setting key 1bd5ec0d665905094c7ff3a7de67edba
Sep 27 06:59:14 dist-test-slave-f892 krb5kdc[19682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:59:14.607916 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:14.588996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40825 (local address 127.0.97.62:37001)
0927 06:59:14.589422 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:14.589428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:14.590489 (+ 1061us) server_negotiation.cc:407] Connection header received
0927 06:59:14.592844 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:14.592849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:14.592945 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:14.593029 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:14.594534 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:14.595706 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:14.597305 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:14.597977 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:14.598139 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:14.602241 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:14.602246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:14.602249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:14.602285 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:14.604750 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:14.605663 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:14.605667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:14.605669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:14.605778 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:14.606283 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:14.606286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:14.606288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:14.606530 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:14.606640 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:14.607505 (+ 865us) server_negotiation.cc:299] Negotiation successful
0927 06:59:14.607676 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":199,"threads_started":1}
I20260927 06:59:14.609310 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:14.589523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37001 (local address 127.0.97.1:40825)
0927 06:59:14.590292 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:14.590353 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:14.592562 (+ 2209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:14.593289 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:14.593297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:14.593764 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:14.594368 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:14.594385 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:14.595898 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:14.595902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:14.597129 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:14.597142 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:14.598200 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:14.598204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:14.598391 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:14.599109 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:14.599158 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:14.601953 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:14.604896 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:14.604907 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:14.604926 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:14.605541 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:14.605888 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:14.605896 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:14.605903 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:14.606166 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:14.606669 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:14.606679 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:14.607088 (+ 409us) client_negotiation.cc:769] Sending connection context
0927 06:59:14.607394 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 06:59:14.607930 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":264,"threads_started":1}
I20260927 06:59:14.610986 19916 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37001
I20260927 06:59:14.611311 19916 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:14.612156 19916 heartbeater.cc:511] Master 127.0.97.62:37001 requested a full tablet report, sending...
I20260927 06:59:14.614632 19727 ts_manager.cc:194] Registered new tserver with Master: 8b2bd73cb8c24a44a4cd033787f44774 (127.0.97.1:38229)
I20260927 06:59:14.616848 19727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40825
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 06:59:14.649960 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37001
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:14.970036 19922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:14.970358 19922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:14.970534 19922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:14.977734 19922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:14.977891 19922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:14.978158 19922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:59:14.987905 19922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:14.989104 19922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:14.990463 19922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:15.004328 19927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:15.004292 19928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:15.005290 19930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:15.005623 19922 server_base.cc:1034] running on GCE node
I20260927 06:59:15.006319 19922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:15.007272 19922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:15.008530 19922 hybrid_clock.cc:648] HybridClock initialized: now 1790492355008473 us; error 52 us; skew 500 ppm
I20260927 06:59:15.008845 19922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:15 dist-test-slave-f892 krb5kdc[19682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:15.012203 19922 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 06:59:15.013743 19922 webserver.cc:460] Webserver started at http://127.0.97.2:35159/ using document root <none> and password file <none>
I20260927 06:59:15.014474 19922 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:15.014606 19922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:15.014923 19922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:15.017874 19922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "fcb45c24e8fe4f75a6dd28397154537c"
format_stamp: "Formatted at 2026-09-27 06:59:15 on dist-test-slave-f892"
server_key: "068a94081f934725ab5773eb88c83e10"
server_key_iv: "102464c7168e0393e5c8930c11c3aaba"
server_key_version: "encryptionkey@0"
I20260927 06:59:15.018764 19922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fcb45c24e8fe4f75a6dd28397154537c"
format_stamp: "Formatted at 2026-09-27 06:59:15 on dist-test-slave-f892"
server_key: "068a94081f934725ab5773eb88c83e10"
server_key_iv: "102464c7168e0393e5c8930c11c3aaba"
server_key_version: "encryptionkey@0"
I20260927 06:59:15.024308 19922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 06:59:15.027727 19937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:15.028918 19922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 06:59:15.029098 19922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "fcb45c24e8fe4f75a6dd28397154537c"
format_stamp: "Formatted at 2026-09-27 06:59:15 on dist-test-slave-f892"
server_key: "068a94081f934725ab5773eb88c83e10"
server_key_iv: "102464c7168e0393e5c8930c11c3aaba"
server_key_version: "encryptionkey@0"
I20260927 06:59:15.029317 19922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:15.044659 19922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:15.048864 19922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:15.049157 19922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:15.050103 19922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:15.051357 19922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:15.051466 19922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:15.051635 19922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:15.051712 19922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:15.085210 19922 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33023
I20260927 06:59:15.085255 20049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33023 every 8 connection(s)
I20260927 06:59:15.086480 19922 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:59:15.092195 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 19922
I20260927 06:59:15.092381 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:59:15.092908 388 external_mini_cluster.cc:1428] Setting key 2ca0be2235b96d0f817d59c1a2e2143a
Sep 27 06:59:15 dist-test-slave-f892 krb5kdc[19682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:59:15.109102 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:15.089068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40269 (local address 127.0.97.62:37001)
0927 06:59:15.089250 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:15.089254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:15.090692 (+ 1438us) server_negotiation.cc:407] Connection header received
0927 06:59:15.092786 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:15.092791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:15.092864 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:15.092957 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:15.094752 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.095507 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.098098 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.098524 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.098620 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:15.102744 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:15.102748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:15.102752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:15.102783 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:15.105836 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:15.106802 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:15.106806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:15.106808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:15.106925 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:15.107451 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:15.107454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:15.107456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:15.107709 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:15.107816 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:15.108691 (+ 875us) server_negotiation.cc:299] Negotiation successful
0927 06:59:15.108863 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 06:59:15.110610 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:15.089546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37001 (local address 127.0.97.2:40269)
0927 06:59:15.090508 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:15.090560 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:15.092463 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:15.093238 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:15.093250 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:15.093929 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:15.094528 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:15.094545 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.095704 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.095712 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:15.097893 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:15.097911 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.098754 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.098763 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:15.099084 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:15.099808 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:15.099841 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:15.102503 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:15.105984 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:15.105997 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:15.106046 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:15.106678 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:15.107040 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:15.107048 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:15.107055 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:15.107327 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:15.107838 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:15.107848 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:15.108290 (+ 442us) client_negotiation.cc:769] Sending connection context
0927 06:59:15.108595 (+ 305us) client_negotiation.cc:240] Negotiation successful
0927 06:59:15.109137 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":324,"threads_started":1}
I20260927 06:59:15.112114 20050 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37001
I20260927 06:59:15.112430 20050 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:15.113175 20050 heartbeater.cc:511] Master 127.0.97.62:37001 requested a full tablet report, sending...
I20260927 06:59:15.114778 19727 ts_manager.cc:194] Registered new tserver with Master: fcb45c24e8fe4f75a6dd28397154537c (127.0.97.2:33023)
I20260927 06:59:15.115628 19727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40269
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 06:59:15.151167 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37001
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:15.473747 20055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:15.474129 20055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:15.474225 20055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:15.481112 20055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:15.481211 20055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:15.481326 20055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:59:15.490998 20055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:15.492269 20055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:15.493482 20055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:15.506446 20061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:15.506486 20060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:15.506460 20063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:15.507627 20055 server_base.cc:1034] running on GCE node
I20260927 06:59:15.508255 20055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:15.509104 20055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:15.510324 20055 hybrid_clock.cc:648] HybridClock initialized: now 1790492355510282 us; error 29 us; skew 500 ppm
I20260927 06:59:15.510622 20055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:15 dist-test-slave-f892 krb5kdc[19682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:15.513665 20055 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 06:59:15.515003 20055 webserver.cc:460] Webserver started at http://127.0.97.3:44775/ using document root <none> and password file <none>
I20260927 06:59:15.515580 20055 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:15.515672 20055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:15.515906 20055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:15.518678 20055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "10ae341e06fb4e6e9ac1f288e473e7f7"
format_stamp: "Formatted at 2026-09-27 06:59:15 on dist-test-slave-f892"
server_key: "a87d6019b36c8943655c43668fbacf58"
server_key_iv: "666c31d15f8ad0a8ad2f5721edeeafd8"
server_key_version: "encryptionkey@0"
I20260927 06:59:15.519385 20055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "10ae341e06fb4e6e9ac1f288e473e7f7"
format_stamp: "Formatted at 2026-09-27 06:59:15 on dist-test-slave-f892"
server_key: "a87d6019b36c8943655c43668fbacf58"
server_key_iv: "666c31d15f8ad0a8ad2f5721edeeafd8"
server_key_version: "encryptionkey@0"
I20260927 06:59:15.524385 20055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 06:59:15.527562 20070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:15.528757 20055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 06:59:15.528872 20055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "10ae341e06fb4e6e9ac1f288e473e7f7"
format_stamp: "Formatted at 2026-09-27 06:59:15 on dist-test-slave-f892"
server_key: "a87d6019b36c8943655c43668fbacf58"
server_key_iv: "666c31d15f8ad0a8ad2f5721edeeafd8"
server_key_version: "encryptionkey@0"
I20260927 06:59:15.529027 20055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:15.577558 20055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:15.581377 20055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:15.581595 20055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:15.582563 20055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:15.583797 20055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:15.583868 20055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:15.583961 20055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:15.584005 20055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:15.618090 20055 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39759
I20260927 06:59:15.618180 20182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39759 every 8 connection(s)
I20260927 06:59:15.619359 20055 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:59:15.619860 19916 heartbeater.cc:503] Master 127.0.97.62:37001 was elected leader, sending a full tablet report...
I20260927 06:59:15.625355 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 20055
I20260927 06:59:15.625545 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:59:15.625995 388 external_mini_cluster.cc:1428] Setting key 82574a339946a3694f76694ca590e572
Sep 27 06:59:15 dist-test-slave-f892 krb5kdc[19682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:59:15.639211 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:15.622131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36571 (local address 127.0.97.62:37001)
0927 06:59:15.622586 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:15.622592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:15.623613 (+ 1021us) server_negotiation.cc:407] Connection header received
0927 06:59:15.625364 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:15.625371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:15.625463 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:15.625596 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:15.627344 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.628141 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.629660 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.630141 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.630278 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:15.633616 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:15.633622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:15.633627 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:15.633672 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:15.636154 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:15.636836 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:15.636841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:15.636846 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:15.637019 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:15.637457 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:15.637464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:15.637467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:15.637772 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:15.637936 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:15.638630 (+ 694us) server_negotiation.cc:299] Negotiation successful
0927 06:59:15.638846 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":210,"threads_started":1}
I20260927 06:59:15.640405 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:15.622808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37001 (local address 127.0.97.3:36571)
0927 06:59:15.623458 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:15.623499 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:15.625039 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:15.625852 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:15.625860 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:15.626460 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:15.627158 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:15.627181 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.628303 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.628309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:15.629512 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:15.629522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.630361 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.630369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:15.630599 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:15.631193 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:15.631213 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:15.633426 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:15.636292 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:15.636300 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:15.636313 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:15.636718 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:15.637135 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:15.637140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:15.637144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:15.637313 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:15.637904 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:15.637911 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:15.638273 (+ 362us) client_negotiation.cc:769] Sending connection context
0927 06:59:15.638543 (+ 270us) client_negotiation.cc:240] Negotiation successful
0927 06:59:15.639008 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":271,"threads_started":1}
I20260927 06:59:15.641726 20183 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37001
I20260927 06:59:15.642035 20183 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:15.642710 20183 heartbeater.cc:511] Master 127.0.97.62:37001 requested a full tablet report, sending...
I20260927 06:59:15.644022 19727 ts_manager.cc:194] Registered new tserver with Master: 10ae341e06fb4e6e9ac1f288e473e7f7 (127.0.97.3:39759)
I20260927 06:59:15.644778 19727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36571
I20260927 06:59:15.653072 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:59:15.668816 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:15.656969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48244 (local address 127.0.97.62:37001)
0927 06:59:15.657125 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:15.657129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:15.657552 (+ 423us) server_negotiation.cc:407] Connection header received
0927 06:59:15.657679 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:15.657682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:15.657738 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:15.657816 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:15.659180 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.659875 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.661161 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.661540 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.661638 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:15.664099 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:15.664103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:15.664106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:15.664134 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:15.666740 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:15.667286 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:15.667290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:15.667292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:15.667405 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:15.667707 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:15.667710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:15.667712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:15.667997 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:15.668162 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:15.668453 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 06:59:15.668606 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 06:59:15.672415 19727 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:59:15.679193 19727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:59:15.704427 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:15.691498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55782 (local address 127.0.97.3:39759)
0927 06:59:15.692826 (+ 1328us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:15.692838 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:15.693851 (+ 1013us) server_negotiation.cc:407] Connection header received
0927 06:59:15.694216 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:15.694228 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:15.694494 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:15.694670 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:15.695877 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.697969 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.701422 (+ 3453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.703014 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.703122 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:15.703343 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:15.703843 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 06:59:15.704019 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1121,"thread_start_us":972,"threads_started":1}
I20260927 06:59:15.704739 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:15.690586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.0.97.1:38229)
0927 06:59:15.693371 (+ 2785us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:15.693381 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:15.693408 (+ 27us) server_negotiation.cc:407] Connection header received
0927 06:59:15.693523 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:15.693531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:15.693815 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:15.694978 (+ 1163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:15.695281 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.697439 (+ 2158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.701979 (+ 4540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.703512 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.703662 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:15.703875 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:15.704194 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 06:59:15.704350 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2596,"thread_start_us":2418,"threads_started":1}
I20260927 06:59:15.704790 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:15.690308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38229 (local address 127.0.0.1:59496)
0927 06:59:15.691173 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:15.691225 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:15.691424 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:15.694153 (+ 2729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:15.694165 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:15.694207 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:15.694701 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:15.694718 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.697654 (+ 2936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.697662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:15.700954 (+ 3292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:15.700975 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.703727 (+ 2752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.703734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:15.703959 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:15.704017 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 06:59:15.704117 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 06:59:15.704314 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":447,"threads_started":1}
I20260927 06:59:15.705890 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:15.690940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39759 (local address 127.0.0.1:55782)
0927 06:59:15.693700 (+ 2760us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:15.693730 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:15.693979 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:15.695064 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:15.695072 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:15.695106 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:15.695635 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:15.695650 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.698210 (+ 2560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.698218 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:15.701191 (+ 2973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:15.701211 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.703306 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.703313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:15.703534 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:15.703598 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 06:59:15.704698 (+ 1100us) client_negotiation.cc:240] Negotiation successful
0927 06:59:15.705043 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2592,"mutex_wait_us":5,"thread_start_us":2424,"threads_started":1}
I20260927 06:59:15.708828 19851 tablet_service.cc:1467] Processing CreateTablet for tablet 52739e21b7274a2ea806a784e1e438f6 (DEFAULT_TABLE table=test-table [id=f4a701ae47064b459540811cafa72a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:15.709275 20118 tablet_service.cc:1467] Processing CreateTablet for tablet 52739e21b7274a2ea806a784e1e438f6 (DEFAULT_TABLE table=test-table [id=f4a701ae47064b459540811cafa72a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:15.710318 19851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52739e21b7274a2ea806a784e1e438f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:15.710631 20118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52739e21b7274a2ea806a784e1e438f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:15.712908 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:15.692466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.0.97.2:33023)
0927 06:59:15.701701 (+ 9235us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:15.701709 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:15.702717 (+ 1008us) server_negotiation.cc:407] Connection header received
0927 06:59:15.702836 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:15.702842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:15.703055 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:15.703244 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:15.704055 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.705479 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.710972 (+ 5493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.711876 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.712035 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:15.712215 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:15.712443 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 06:59:15.712597 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9079,"thread_start_us":8948,"threads_started":1}
I20260927 06:59:15.713443 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:15.691632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33023 (local address 127.0.0.1:36346)
0927 06:59:15.702451 (+ 10819us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:15.702474 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:15.702608 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:15.703421 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:15.703425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:15.703449 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:15.703874 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:15.703886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.705631 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.705635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:15.710814 (+ 5179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:15.710827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.712076 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.712082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:15.712247 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:15.712278 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 06:59:15.712392 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 06:59:15.712512 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10704,"thread_start_us":10567,"threads_started":1}
I20260927 06:59:15.716365 19985 tablet_service.cc:1467] Processing CreateTablet for tablet 52739e21b7274a2ea806a784e1e438f6 (DEFAULT_TABLE table=test-table [id=f4a701ae47064b459540811cafa72a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:15.717481 19985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52739e21b7274a2ea806a784e1e438f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:15.725106 20202 tablet_bootstrap.cc:492] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7: Bootstrap starting.
I20260927 06:59:15.728767 20202 tablet_bootstrap.cc:654] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:15.729602 20203 tablet_bootstrap.cc:492] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774: Bootstrap starting.
I20260927 06:59:15.730085 20202 log.cc:824] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:15.732162 20204 tablet_bootstrap.cc:492] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c: Bootstrap starting.
I20260927 06:59:15.733332 20203 tablet_bootstrap.cc:654] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:15.734576 20202 tablet_bootstrap.cc:492] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7: No bootstrap required, opened a new log
I20260927 06:59:15.734587 20203 log.cc:824] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:15.734845 20202 ts_tablet_manager.cc:1397] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 06:59:15.735595 20204 tablet_bootstrap.cc:654] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:15.736780 20204 log.cc:824] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:15.737771 20203 tablet_bootstrap.cc:492] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774: No bootstrap required, opened a new log
I20260927 06:59:15.738094 20203 ts_tablet_manager.cc:1397] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260927 06:59:15.739853 20204 tablet_bootstrap.cc:492] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c: No bootstrap required, opened a new log
I20260927 06:59:15.740104 20204 ts_tablet_manager.cc:1397] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260927 06:59:15.745134 20202 raft_consensus.cc:348] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2bd73cb8c24a44a4cd033787f44774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38229 } } peers { permanent_uuid: "fcb45c24e8fe4f75a6dd28397154537c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "10ae341e06fb4e6e9ac1f288e473e7f7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39759 } }
I20260927 06:59:15.745577 20202 raft_consensus.cc:374] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:15.745677 20202 raft_consensus.cc:729] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ae341e06fb4e6e9ac1f288e473e7f7, State: Initialized, Role: FOLLOWER
I20260927 06:59:15.745445 20203 raft_consensus.cc:348] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2bd73cb8c24a44a4cd033787f44774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38229 } } peers { permanent_uuid: "fcb45c24e8fe4f75a6dd28397154537c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "10ae341e06fb4e6e9ac1f288e473e7f7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39759 } }
I20260927 06:59:15.745862 20203 raft_consensus.cc:374] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:15.746024 20203 raft_consensus.cc:729] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2bd73cb8c24a44a4cd033787f44774, State: Initialized, Role: FOLLOWER
I20260927 06:59:15.746284 20202 consensus_queue.cc:260] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2bd73cb8c24a44a4cd033787f44774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38229 } } peers { permanent_uuid: "fcb45c24e8fe4f75a6dd28397154537c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "10ae341e06fb4e6e9ac1f288e473e7f7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39759 } }
I20260927 06:59:15.746613 20203 consensus_queue.cc:260] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2bd73cb8c24a44a4cd033787f44774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38229 } } peers { permanent_uuid: "fcb45c24e8fe4f75a6dd28397154537c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "10ae341e06fb4e6e9ac1f288e473e7f7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39759 } }
I20260927 06:59:15.747612 20183 heartbeater.cc:503] Master 127.0.97.62:37001 was elected leader, sending a full tablet report...
I20260927 06:59:15.748059 20202 ts_tablet_manager.cc:1428] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260927 06:59:15.750015 20204 raft_consensus.cc:348] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2bd73cb8c24a44a4cd033787f44774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38229 } } peers { permanent_uuid: "fcb45c24e8fe4f75a6dd28397154537c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "10ae341e06fb4e6e9ac1f288e473e7f7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39759 } }
I20260927 06:59:15.750444 20204 raft_consensus.cc:374] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:15.750571 20204 raft_consensus.cc:729] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb45c24e8fe4f75a6dd28397154537c, State: Initialized, Role: FOLLOWER
I20260927 06:59:15.751099 20204 consensus_queue.cc:260] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2bd73cb8c24a44a4cd033787f44774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38229 } } peers { permanent_uuid: "fcb45c24e8fe4f75a6dd28397154537c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "10ae341e06fb4e6e9ac1f288e473e7f7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39759 } }
I20260927 06:59:15.752194 20050 heartbeater.cc:503] Master 127.0.97.62:37001 was elected leader, sending a full tablet report...
I20260927 06:59:15.752794 20204 ts_tablet_manager.cc:1428] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260927 06:59:15.756312 20203 ts_tablet_manager.cc:1428] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774: Time spent starting tablet: real 0.018s user 0.008s sys 0.007s
W20260927 06:59:15.838557 20051 tablet.cc:2369] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:59:15.839203 19917 tablet.cc:2369] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:59:15.871425 20184 tablet.cc:2369] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:59:16.000114 20208 raft_consensus.cc:482] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:59:16.000432 20208 raft_consensus.cc:504] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2bd73cb8c24a44a4cd033787f44774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38229 } } peers { permanent_uuid: "fcb45c24e8fe4f75a6dd28397154537c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "10ae341e06fb4e6e9ac1f288e473e7f7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39759 } }
I20260927 06:59:16.002156 20208 leader_election.cc:288] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b2bd73cb8c24a44a4cd033787f44774 (127.0.97.1:38229), fcb45c24e8fe4f75a6dd28397154537c (127.0.97.2:33023)
I20260927 06:59:16.009009 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:16.002605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37503 (local address 127.0.97.1:38229)
0927 06:59:16.002779 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:16.002785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:16.002802 (+ 17us) server_negotiation.cc:407] Connection header received
0927 06:59:16.002873 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:16.002878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:16.002945 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:16.003035 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:16.003889 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:16.004867 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:16.007246 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:16.008123 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:16.008279 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:16.008380 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:16.008643 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 06:59:16.008763 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 06:59:16.009009 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:16.002319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38229 (local address 127.0.97.3:37503)
0927 06:59:16.002529 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:16.002548 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:16.002665 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:16.003397 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:16.003405 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:16.003444 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:16.003717 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:16.003727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:16.005009 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:16.005015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:16.007067 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:16.007080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:16.008285 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:16.008291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:16.008473 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:16.008496 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 06:59:16.008615 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 06:59:16.008737 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":1}
I20260927 06:59:16.010052 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:16.002738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33023 (local address 127.0.97.3:49157)
0927 06:59:16.003903 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:16.003920 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:16.004051 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:16.004399 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:16.004403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:16.004426 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:16.004737 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:16.004746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:16.005759 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:16.005764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:16.007221 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:16.007235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:16.008957 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:16.008963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:16.009471 (+ 508us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:16.009521 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 06:59:16.009612 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 06:59:16.009731 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"thread_start_us":855,"threads_started":1}
I20260927 06:59:16.010082 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:16.003020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49157 (local address 127.0.97.2:33023)
0927 06:59:16.003276 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:16.003284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:16.004015 (+ 731us) server_negotiation.cc:407] Connection header received
0927 06:59:16.004179 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:16.004186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:16.004271 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:16.004362 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:16.004887 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:16.005605 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:16.007427 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:16.008619 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:16.008739 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:16.008822 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:16.009650 (+ 828us) server_negotiation.cc:299] Negotiation successful
0927 06:59:16.009785 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 06:59:16.010049 19871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52739e21b7274a2ea806a784e1e438f6" candidate_uuid: "10ae341e06fb4e6e9ac1f288e473e7f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b2bd73cb8c24a44a4cd033787f44774" is_pre_election: true
I20260927 06:59:16.010586 19871 raft_consensus.cc:2454] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ae341e06fb4e6e9ac1f288e473e7f7 in term 0.
I20260927 06:59:16.011250 20071 leader_election.cc:302] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ae341e06fb4e6e9ac1f288e473e7f7, 8b2bd73cb8c24a44a4cd033787f44774; no voters:
I20260927 06:59:16.011674 20208 raft_consensus.cc:2781] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:59:16.011814 20208 raft_consensus.cc:482] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:59:16.011905 20208 raft_consensus.cc:3037] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:16.011831 20005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52739e21b7274a2ea806a784e1e438f6" candidate_uuid: "10ae341e06fb4e6e9ac1f288e473e7f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcb45c24e8fe4f75a6dd28397154537c" is_pre_election: true
I20260927 06:59:16.012365 20005 raft_consensus.cc:2454] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ae341e06fb4e6e9ac1f288e473e7f7 in term 0.
I20260927 06:59:16.015115 20208 raft_consensus.cc:504] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2bd73cb8c24a44a4cd033787f44774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38229 } } peers { permanent_uuid: "fcb45c24e8fe4f75a6dd28397154537c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "10ae341e06fb4e6e9ac1f288e473e7f7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39759 } }
I20260927 06:59:16.015797 20208 leader_election.cc:288] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [CANDIDATE]: Term 1 election: Requested vote from peers 8b2bd73cb8c24a44a4cd033787f44774 (127.0.97.1:38229), fcb45c24e8fe4f75a6dd28397154537c (127.0.97.2:33023)
I20260927 06:59:16.016193 19871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52739e21b7274a2ea806a784e1e438f6" candidate_uuid: "10ae341e06fb4e6e9ac1f288e473e7f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b2bd73cb8c24a44a4cd033787f44774"
I20260927 06:59:16.016314 20005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52739e21b7274a2ea806a784e1e438f6" candidate_uuid: "10ae341e06fb4e6e9ac1f288e473e7f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcb45c24e8fe4f75a6dd28397154537c"
I20260927 06:59:16.016407 19871 raft_consensus.cc:3037] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:16.016577 20005 raft_consensus.cc:3037] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:16.018703 19871 raft_consensus.cc:2454] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ae341e06fb4e6e9ac1f288e473e7f7 in term 1.
I20260927 06:59:16.018836 20005 raft_consensus.cc:2454] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ae341e06fb4e6e9ac1f288e473e7f7 in term 1.
I20260927 06:59:16.019119 20071 leader_election.cc:302] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ae341e06fb4e6e9ac1f288e473e7f7, 8b2bd73cb8c24a44a4cd033787f44774; no voters:
I20260927 06:59:16.019407 20208 raft_consensus.cc:2781] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:16.019870 20208 raft_consensus.cc:686] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [term 1 LEADER]: Becoming Leader. State: Replica: 10ae341e06fb4e6e9ac1f288e473e7f7, State: Running, Role: LEADER
I20260927 06:59:16.020358 20208 consensus_queue.cc:237] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2bd73cb8c24a44a4cd033787f44774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38229 } } peers { permanent_uuid: "fcb45c24e8fe4f75a6dd28397154537c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "10ae341e06fb4e6e9ac1f288e473e7f7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39759 } }
I20260927 06:59:16.025131 19727 catalog_manager.cc:5495] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ae341e06fb4e6e9ac1f288e473e7f7 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "10ae341e06fb4e6e9ac1f288e473e7f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2bd73cb8c24a44a4cd033787f44774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcb45c24e8fe4f75a6dd28397154537c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ae341e06fb4e6e9ac1f288e473e7f7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39759 } health_report { overall_health: HEALTHY } } }
I20260927 06:59:16.078061 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:16.072757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55790 (local address 127.0.97.3:39759)
0927 06:59:16.072960 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:16.072965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:16.072983 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:59:16.073182 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:16.073186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:16.073260 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:16.073379 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:16.073758 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:16.074559 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:16.076014 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:16.076390 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:16.076488 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:16.076942 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:16.077407 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:16.077537 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:16.077660 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 06:59:16.077722 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 06:59:16.085667 20005 raft_consensus.cc:1261] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c [term 1 FOLLOWER]: Refusing update from remote peer 10ae341e06fb4e6e9ac1f288e473e7f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:16.085659 19871 raft_consensus.cc:1261] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774 [term 1 FOLLOWER]: Refusing update from remote peer 10ae341e06fb4e6e9ac1f288e473e7f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:16.086661 20212 consensus_queue.cc:1035] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcb45c24e8fe4f75a6dd28397154537c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:16.087132 20208 consensus_queue.cc:1035] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b2bd73cb8c24a44a4cd033787f44774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:16.104758 20216 mvcc.cc:205] Tried to move back new op lower bound from 7333856690516606976 to 7333856690263150592. Current Snapshot: MvccSnapshot[applied={T|T < 7333856690516606976}]
I20260927 06:59:16.113945 20215 mvcc.cc:205] Tried to move back new op lower bound from 7333856690516606976 to 7333856690263150592. Current Snapshot: MvccSnapshot[applied={T|T < 7333856690516606976 or (T in {7333856690516606976})}]
I20260927 06:59:16.124431 19727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:16.124653 19727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:16.128098 19727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3956ec59bd8242eab20cdb5f728b058c: Sending DeleteTablet for 3 replicas of tablet 52739e21b7274a2ea806a784e1e438f6
I20260927 06:59:16.129142 19851 tablet_service.cc:1514] Processing DeleteTablet for tablet 52739e21b7274a2ea806a784e1e438f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59496
I20260927 06:59:16.129359 20118 tablet_service.cc:1514] Processing DeleteTablet for tablet 52739e21b7274a2ea806a784e1e438f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55782
I20260927 06:59:16.129346 19985 tablet_service.cc:1514] Processing DeleteTablet for tablet 52739e21b7274a2ea806a784e1e438f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36346
I20260927 06:59:16.131603 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 19788
I20260927 06:59:16.141192 20226 tablet_replica.cc:330] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7: stopping tablet replica
I20260927 06:59:16.141940 20226 raft_consensus.cc:2229] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:59:16.142858 20226 raft_consensus.cc:2258] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:16.143985 20227 tablet_replica.cc:330] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c: stopping tablet replica
I20260927 06:59:16.144507 20227 raft_consensus.cc:2229] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:16.145095 20227 raft_consensus.cc:2258] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:16.145148 20226 ts_tablet_manager.cc:1905] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:16.146938 20225 tablet_replica.cc:330] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774: stopping tablet replica
I20260927 06:59:16.147413 20225 raft_consensus.cc:2229] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:16.147979 20225 raft_consensus.cc:2258] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:16.150118 20225 ts_tablet_manager.cc:1905] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:16.152654 20226 ts_tablet_manager.cc:1918] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:16.152820 20226 log.cc:1196] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/52739e21b7274a2ea806a784e1e438f6
I20260927 06:59:16.153364 20226 ts_tablet_manager.cc:1939] T 52739e21b7274a2ea806a784e1e438f6 P 10ae341e06fb4e6e9ac1f288e473e7f7: Deleting consensus metadata
I20260927 06:59:16.155285 19714 catalog_manager.cc:4841] TS 10ae341e06fb4e6e9ac1f288e473e7f7 (127.0.97.3:39759): tablet 52739e21b7274a2ea806a784e1e438f6 (table test-table [id=f4a701ae47064b459540811cafa72a44]) successfully deleted
I20260927 06:59:16.156355 20227 ts_tablet_manager.cc:1905] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 06:59:16 dist-test-slave-f892 krb5kdc[19682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 06:59:16.159150 20225 ts_tablet_manager.cc:1918] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:16.159380 20225 log.cc:1196] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/52739e21b7274a2ea806a784e1e438f6
I20260927 06:59:16.160377 20225 ts_tablet_manager.cc:1939] T 52739e21b7274a2ea806a784e1e438f6 P 8b2bd73cb8c24a44a4cd033787f44774: Deleting consensus metadata
I20260927 06:59:16.162168 19711 catalog_manager.cc:4841] TS 8b2bd73cb8c24a44a4cd033787f44774 (127.0.97.1:38229): tablet 52739e21b7274a2ea806a784e1e438f6 (table test-table [id=f4a701ae47064b459540811cafa72a44]) successfully deleted
I20260927 06:59:16.163076 20227 ts_tablet_manager.cc:1918] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:16.163232 20227 log.cc:1196] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/52739e21b7274a2ea806a784e1e438f6
I20260927 06:59:16.163739 20227 ts_tablet_manager.cc:1939] T 52739e21b7274a2ea806a784e1e438f6 P fcb45c24e8fe4f75a6dd28397154537c: Deleting consensus metadata
I20260927 06:59:16.165246 19711 catalog_manager.cc:4841] TS fcb45c24e8fe4f75a6dd28397154537c (127.0.97.2:33023): tablet 52739e21b7274a2ea806a784e1e438f6 (table test-table [id=f4a701ae47064b459540811cafa72a44]) successfully deleted
I20260927 06:59:16.165716 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:16.134091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.0.97.1:38229)
0927 06:59:16.134316 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:16.134325 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:16.146244 (+ 11919us) server_negotiation.cc:407] Connection header received
0927 06:59:16.146452 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:16.146463 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:16.146569 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:16.146695 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:16.148495 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:16.149623 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:16.151464 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:16.152123 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:16.152223 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:16.157597 (+ 5374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:16.157611 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:16.157628 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:16.157669 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:16.161780 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:16.163001 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:16.163010 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:16.163014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:16.163143 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:16.163583 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:16.163588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:16.163591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:16.164062 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:16.164357 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:16.164743 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 06:59:16.164960 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 06:59:16.166424 19911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:16.254812 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 19788
I20260927 06:59:16.274261 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 19922
Sep 27 06:59:16 dist-test-slave-f892 krb5kdc[19682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 06:59:16.290812 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:16.275086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36356 (local address 127.0.97.2:33023)
0927 06:59:16.275282 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:16.275288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:16.275311 (+ 23us) server_negotiation.cc:407] Connection header received
0927 06:59:16.275424 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:16.275430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:16.275535 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:16.275648 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:16.276910 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:16.278081 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:16.279369 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:16.280004 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:16.280142 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:16.284118 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:16.284137 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:16.284156 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:16.284197 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:16.287882 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:16.288570 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:16.288580 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:16.288584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:16.288738 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:16.289137 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:16.289142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:16.289146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:16.289613 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:16.289898 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:16.290216 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 06:59:16.290434 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 06:59:16.291502 20045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:16.379594 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 19922
I20260927 06:59:16.397289 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 20055
Sep 27 06:59:16 dist-test-slave-f892 krb5kdc[19682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 06:59:16.415369 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:16.398134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55792 (local address 127.0.97.3:39759)
0927 06:59:16.398343 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:16.398349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:16.398372 (+ 23us) server_negotiation.cc:407] Connection header received
0927 06:59:16.398538 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:16.398550 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:16.398642 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:16.398756 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:16.400021 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:16.401296 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:16.402764 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:16.403475 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:16.403613 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:16.408055 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:16.408071 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:16.408098 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:16.408162 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:16.412062 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:16.412848 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:16.412863 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:16.412866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:16.413087 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:16.413519 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:16.413525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:16.413528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:16.414066 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:16.414368 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:16.414702 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 06:59:16.414950 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 06:59:16.416190 20178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:16.535439 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 20055
I20260927 06:59:16.555001 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 19695
I20260927 06:59:16.556082 19757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:16.664873 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 19695
2026-09-27T06:59:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4420 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T06:59:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:16Z Disabled control of system clock
I20260927 06:59:16.739509 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34877
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 06:59:17.058732 20236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:17.059047 20236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:17.059126 20236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:17.066001 20236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:17.066108 20236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:17.066144 20236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:17.066195 20236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:17.066256 20236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:17.074904 20236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34877
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:17.075973 20236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:17.077775 20236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:17.089226 20241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:17.089406 20242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:17.089370 20244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:17.089915 20236 server_base.cc:1034] running on GCE node
I20260927 06:59:17.090652 20236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:17.091992 20236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:17.093185 20236 hybrid_clock.cc:648] HybridClock initialized: now 1790492357093131 us; error 53 us; skew 500 ppm
I20260927 06:59:17.093477 20236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:59:17.095654 20236 webserver.cc:460] Webserver started at http://127.0.97.62:39583/ using document root <none> and password file <none>
I20260927 06:59:17.096253 20236 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:17.096334 20236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:17.096591 20236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:17.099517 20236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "3378e67ace7f4e0694db3aac49fed038"
format_stamp: "Formatted at 2026-09-27 06:59:17 on dist-test-slave-f892"
server_key: "fe6876b9e6fbde74bcb63ae9d107bcbe"
server_key_iv: "d20d2e1fdd14cfd2d392668263732c2c"
server_key_version: "encryptionkey@0"
I20260927 06:59:17.100230 20236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "3378e67ace7f4e0694db3aac49fed038"
format_stamp: "Formatted at 2026-09-27 06:59:17 on dist-test-slave-f892"
server_key: "fe6876b9e6fbde74bcb63ae9d107bcbe"
server_key_iv: "d20d2e1fdd14cfd2d392668263732c2c"
server_key_version: "encryptionkey@0"
I20260927 06:59:17.104988 20236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260927 06:59:17.108240 20250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:17.109556 20236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 06:59:17.109764 20236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "3378e67ace7f4e0694db3aac49fed038"
format_stamp: "Formatted at 2026-09-27 06:59:17 on dist-test-slave-f892"
server_key: "fe6876b9e6fbde74bcb63ae9d107bcbe"
server_key_iv: "d20d2e1fdd14cfd2d392668263732c2c"
server_key_version: "encryptionkey@0"
I20260927 06:59:17.109969 20236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:17.125313 20236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:17.126168 20236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:17.126451 20236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:17.144830 20236 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34877
I20260927 06:59:17.144863 20301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34877 every 8 connection(s)
I20260927 06:59:17.146147 20236 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:17.149645 20302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:17.150410 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 20236
I20260927 06:59:17.150593 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:17.151048 388 external_mini_cluster.cc:1428] Setting key d4425c93ccd1f45e969c10c3fb2d9694
I20260927 06:59:17.162657 20302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038: Bootstrap starting.
I20260927 06:59:17.164149 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:17.152899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41526 (local address 127.0.97.62:34877)
0927 06:59:17.153838 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:17.153848 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:17.153885 (+ 37us) server_negotiation.cc:407] Connection header received
0927 06:59:17.155533 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:17.155552 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:17.155962 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:17.156257 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:17.156824 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:17.158083 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:17.159996 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:17.160666 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:17.160793 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:17.161986 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:17.161993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:17.162054 (+ 61us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:17.162539 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:17.162609 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:17.162627 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:17.162709 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:17.162993 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 06:59:17.163302 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":370,"threads_started":1}
I20260927 06:59:17.165747 20302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:17.166870 20302 log.cc:824] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:17.169461 20302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038: No bootstrap required, opened a new log
I20260927 06:59:17.176955 20302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3378e67ace7f4e0694db3aac49fed038" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34877 } }
I20260927 06:59:17.177291 20302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:17.177390 20302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3378e67ace7f4e0694db3aac49fed038, State: Initialized, Role: FOLLOWER
I20260927 06:59:17.177975 20302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3378e67ace7f4e0694db3aac49fed038" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34877 } }
I20260927 06:59:17.178267 20302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:17.178392 20302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:17.178570 20302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:17.180742 20302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3378e67ace7f4e0694db3aac49fed038" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34877 } }
I20260927 06:59:17.181272 20302 leader_election.cc:302] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3378e67ace7f4e0694db3aac49fed038; no voters:
I20260927 06:59:17.181810 20302 leader_election.cc:288] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:17.181957 20307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:17.183238 20307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038 [term 1 LEADER]: Becoming Leader. State: Replica: 3378e67ace7f4e0694db3aac49fed038, State: Running, Role: LEADER
I20260927 06:59:17.183712 20307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3378e67ace7f4e0694db3aac49fed038" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34877 } }
I20260927 06:59:17.183915 20302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:17.186105 20309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3378e67ace7f4e0694db3aac49fed038. Latest consensus state: current_term: 1 leader_uuid: "3378e67ace7f4e0694db3aac49fed038" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3378e67ace7f4e0694db3aac49fed038" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34877 } } }
I20260927 06:59:17.186480 20309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:17.186342 20308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3378e67ace7f4e0694db3aac49fed038" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3378e67ace7f4e0694db3aac49fed038" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34877 } } }
I20260927 06:59:17.186725 20308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:17.188634 20313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:17.193274 20313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:17.201669 20313 catalog_manager.cc:1347] Generated new cluster ID: d862396eb7a044b89880f6ff178b97fa
I20260927 06:59:17.201771 20313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:17.234545 20313 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:17.236301 20313 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:17.242525 20313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038: Generated new TSK 0
I20260927 06:59:17.243305 20313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:17.253612 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34877
--builtin_ntp_servers=127.0.97.20:38581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 06:59:17.581278 20326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:17.581524 20326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:17.581694 20326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:17.588778 20326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:17.588927 20326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:17.589107 20326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 06:59:17.597808 20326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:17.598995 20326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:17.600667 20326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:17.613659 20331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:17.613584 20334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:17.613633 20332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:17.614316 20326 server_base.cc:1034] running on GCE node
I20260927 06:59:17.615015 20326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:17.615950 20326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:17.617194 20326 hybrid_clock.cc:648] HybridClock initialized: now 1790492357617153 us; error 56 us; skew 500 ppm
I20260927 06:59:17.617518 20326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:59:17.619768 20326 webserver.cc:460] Webserver started at http://127.0.97.1:33651/ using document root <none> and password file <none>
I20260927 06:59:17.620457 20326 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:17.620591 20326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:17.620913 20326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:17.623697 20326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-0/data/instance:
uuid: "de006593ff774524b888b71d481056db"
format_stamp: "Formatted at 2026-09-27 06:59:17 on dist-test-slave-f892"
server_key: "42a7006e3b30db0a8ddb6dec92b2368f"
server_key_iv: "59ea7ae840efb456d079cfe54fba4341"
server_key_version: "encryptionkey@0"
I20260927 06:59:17.624464 20326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-0/wal/instance:
uuid: "de006593ff774524b888b71d481056db"
format_stamp: "Formatted at 2026-09-27 06:59:17 on dist-test-slave-f892"
server_key: "42a7006e3b30db0a8ddb6dec92b2368f"
server_key_iv: "59ea7ae840efb456d079cfe54fba4341"
server_key_version: "encryptionkey@0"
I20260927 06:59:17.629197 20326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 06:59:17.632459 20340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:17.633687 20326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 06:59:17.633862 20326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-0/wal
uuid: "de006593ff774524b888b71d481056db"
format_stamp: "Formatted at 2026-09-27 06:59:17 on dist-test-slave-f892"
server_key: "42a7006e3b30db0a8ddb6dec92b2368f"
server_key_iv: "59ea7ae840efb456d079cfe54fba4341"
server_key_version: "encryptionkey@0"
I20260927 06:59:17.634106 20326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:17.656239 20326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:17.657126 20326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:17.657419 20326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:17.658456 20326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:17.659955 20326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:17.660080 20326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:17.660211 20326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:17.660317 20326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:17.701040 20326 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40107
I20260927 06:59:17.701134 20452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40107 every 8 connection(s)
I20260927 06:59:17.702407 20326 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-0/data/info.pb
I20260927 06:59:17.705281 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 20326
I20260927 06:59:17.705436 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-0/wal/instance
I20260927 06:59:17.705839 388 external_mini_cluster.cc:1428] Setting key 688d2a44111af120a7f147c6b8981ca5
I20260927 06:59:17.708667 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34877
--builtin_ntp_servers=127.0.97.20:38581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 06:59:17.716102 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:17.704950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51729 (local address 127.0.97.62:34877)
0927 06:59:17.705465 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:17.705471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:17.706807 (+ 1336us) server_negotiation.cc:407] Connection header received
0927 06:59:17.708824 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:17.708831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:17.708919 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:17.709031 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:17.710058 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:17.711220 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:17.712741 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:17.713503 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:17.713662 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:17.715129 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:17.715138 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:17.715145 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:17.715171 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:17.715223 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:17.715240 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:17.715336 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:17.715543 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 06:59:17.715657 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":267,"threads_started":1}
I20260927 06:59:17.716928 20456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:17.705556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34877 (local address 127.0.97.1:51729)
0927 06:59:17.706643 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:17.706693 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:17.708555 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:17.709217 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:17.709225 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:17.709312 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:17.709860 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:17.709875 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:17.711368 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:17.711372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:17.712585 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:17.712597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:17.713746 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:17.713753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:17.713974 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:17.714806 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:59:17.714808 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:17.714847 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:59:17.714924 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:17.715340 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:17.715348 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:17.715354 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 06:59:17.715491 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 06:59:17.715777 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":426,"threads_started":1}
I20260927 06:59:17.718545 20453 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34877
I20260927 06:59:17.718847 20453 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:17.719542 20453 heartbeater.cc:511] Master 127.0.97.62:34877 requested a full tablet report, sending...
I20260927 06:59:17.721519 20267 ts_manager.cc:194] Registered new tserver with Master: de006593ff774524b888b71d481056db (127.0.97.1:40107)
I20260927 06:59:17.723008 20267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:51729
W20260927 06:59:18.028070 20457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:18.028398 20457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:18.028560 20457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:18.035975 20457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:18.036110 20457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:18.036303 20457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 06:59:18.045567 20457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:18.046849 20457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:18.048578 20457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:18.061630 20462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:18.061726 20463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:18.061764 20465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:18.067344 20457 server_base.cc:1034] running on GCE node
I20260927 06:59:18.068004 20457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:18.068904 20457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:18.070116 20457 hybrid_clock.cc:648] HybridClock initialized: now 1790492358070083 us; error 52 us; skew 500 ppm
I20260927 06:59:18.070438 20457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:59:18.072575 20457 webserver.cc:460] Webserver started at http://127.0.97.2:39033/ using document root <none> and password file <none>
I20260927 06:59:18.073280 20457 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:18.073400 20457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:18.073673 20457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:18.076416 20457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-1/data/instance:
uuid: "fbfce91c028c48fdbf430f37f7734ebc"
format_stamp: "Formatted at 2026-09-27 06:59:18 on dist-test-slave-f892"
server_key: "e9fc55bab63fbadae00e9982afe02bf9"
server_key_iv: "7fdff848b3193c1a86e8bb67330df88c"
server_key_version: "encryptionkey@0"
I20260927 06:59:18.077119 20457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fbfce91c028c48fdbf430f37f7734ebc"
format_stamp: "Formatted at 2026-09-27 06:59:18 on dist-test-slave-f892"
server_key: "e9fc55bab63fbadae00e9982afe02bf9"
server_key_iv: "7fdff848b3193c1a86e8bb67330df88c"
server_key_version: "encryptionkey@0"
I20260927 06:59:18.082109 20457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 06:59:18.085318 20471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:18.086596 20457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:59:18.086781 20457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-1/wal
uuid: "fbfce91c028c48fdbf430f37f7734ebc"
format_stamp: "Formatted at 2026-09-27 06:59:18 on dist-test-slave-f892"
server_key: "e9fc55bab63fbadae00e9982afe02bf9"
server_key_iv: "7fdff848b3193c1a86e8bb67330df88c"
server_key_version: "encryptionkey@0"
I20260927 06:59:18.086997 20457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:18.104130 20457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:18.104918 20457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:18.105214 20457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:18.106199 20457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:18.107589 20457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:18.107720 20457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:18.107854 20457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:18.107944 20457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:18.139750 20457 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36141
I20260927 06:59:18.139837 20583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36141 every 8 connection(s)
I20260927 06:59:18.141011 20457 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-1/data/info.pb
I20260927 06:59:18.144915 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 20457
I20260927 06:59:18.145076 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-1/wal/instance
I20260927 06:59:18.145574 388 external_mini_cluster.cc:1428] Setting key c3d67f909c1590f0ca24b3a885ca01d3
I20260927 06:59:18.148798 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34877
--builtin_ntp_servers=127.0.97.20:38581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 06:59:18.153751 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.143444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49803 (local address 127.0.97.62:34877)
0927 06:59:18.143628 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:18.143633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:18.145141 (+ 1508us) server_negotiation.cc:407] Connection header received
0927 06:59:18.147378 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:18.147383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:18.147463 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:18.147577 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:18.148649 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.149498 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.151099 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.151489 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.151634 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.152931 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:18.152936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:18.152939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:18.152955 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:18.152997 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:18.153009 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:18.153128 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:18.153343 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 06:59:18.153435 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 06:59:18.154870 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.143902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34877 (local address 127.0.97.2:49803)
0927 06:59:18.144928 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:18.144976 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:18.147063 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:18.147891 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:18.147898 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:18.147977 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:18.148484 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.148498 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.149652 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.149658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.150942 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.150953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.151707 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.151713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.151935 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.152618 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:59:18.152621 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:18.152655 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:59:18.152726 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:18.153128 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:18.153140 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:18.153150 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 06:59:18.153332 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 06:59:18.153669 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":367,"threads_started":1}
I20260927 06:59:18.156138 20584 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34877
I20260927 06:59:18.156435 20584 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:18.157109 20584 heartbeater.cc:511] Master 127.0.97.62:34877 requested a full tablet report, sending...
I20260927 06:59:18.158550 20267 ts_manager.cc:194] Registered new tserver with Master: fbfce91c028c48fdbf430f37f7734ebc (127.0.97.2:36141)
I20260927 06:59:18.159400 20267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:49803
W20260927 06:59:18.473031 20587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:18.473313 20587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:18.473397 20587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:18.480849 20587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:18.480937 20587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:18.481040 20587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 06:59:18.489778 20587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:18.491067 20587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:18.492865 20587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:18.505785 20592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:18.505822 20593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:18.505894 20595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:18.506661 20587 server_base.cc:1034] running on GCE node
I20260927 06:59:18.507308 20587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:18.508190 20587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:18.509446 20587 hybrid_clock.cc:648] HybridClock initialized: now 1790492358509392 us; error 42 us; skew 500 ppm
I20260927 06:59:18.509770 20587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:59:18.512042 20587 webserver.cc:460] Webserver started at http://127.0.97.3:41769/ using document root <none> and password file <none>
I20260927 06:59:18.512727 20587 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:18.512864 20587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:18.513253 20587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:18.516122 20587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-2/data/instance:
uuid: "243bd786801848c6b694b0c593101d08"
format_stamp: "Formatted at 2026-09-27 06:59:18 on dist-test-slave-f892"
server_key: "979bc302129d0b1e8b29ac3a84f6e8f2"
server_key_iv: "ccbd00886464470b876a9a7b21e5a2eb"
server_key_version: "encryptionkey@0"
I20260927 06:59:18.516923 20587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-2/wal/instance:
uuid: "243bd786801848c6b694b0c593101d08"
format_stamp: "Formatted at 2026-09-27 06:59:18 on dist-test-slave-f892"
server_key: "979bc302129d0b1e8b29ac3a84f6e8f2"
server_key_iv: "ccbd00886464470b876a9a7b21e5a2eb"
server_key_version: "encryptionkey@0"
I20260927 06:59:18.521817 20587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260927 06:59:18.525015 20601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:18.526377 20587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:59:18.526505 20587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-2/wal
uuid: "243bd786801848c6b694b0c593101d08"
format_stamp: "Formatted at 2026-09-27 06:59:18 on dist-test-slave-f892"
server_key: "979bc302129d0b1e8b29ac3a84f6e8f2"
server_key_iv: "ccbd00886464470b876a9a7b21e5a2eb"
server_key_version: "encryptionkey@0"
I20260927 06:59:18.526671 20587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:18.541177 20587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:18.541880 20587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:18.542178 20587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:18.543040 20587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:18.544358 20587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:18.544443 20587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:18.544520 20587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:18.544572 20587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:18.577102 20587 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42199
I20260927 06:59:18.577158 20713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42199 every 8 connection(s)
I20260927 06:59:18.578431 20587 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-2/data/info.pb
I20260927 06:59:18.588433 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 20587
I20260927 06:59:18.588615 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-2/wal/instance
I20260927 06:59:18.589048 388 external_mini_cluster.cc:1428] Setting key bdb1e92838b72134a1038610aedcc2d8
I20260927 06:59:18.590967 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.580914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52929 (local address 127.0.97.62:34877)
0927 06:59:18.581134 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:18.581140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:18.582736 (+ 1596us) server_negotiation.cc:407] Connection header received
0927 06:59:18.584600 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:18.584605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:18.584694 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:18.584818 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:18.585747 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.586511 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.588016 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.588476 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.588622 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.590073 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:18.590079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:18.590084 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:18.590109 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:18.590138 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:18.590150 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:18.590266 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:18.590520 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 06:59:18.590613 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 06:59:18.591940 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.581420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34877 (local address 127.0.97.3:52929)
0927 06:59:18.582559 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:18.582612 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:18.584332 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:18.585011 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:18.585019 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:18.585096 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:18.585606 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.585620 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.586652 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.586655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.587881 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.587893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.588722 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.588730 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.588942 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.589681 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:59:18.589684 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:18.589723 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:59:18.589831 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:18.590294 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:18.590321 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:18.590333 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 06:59:18.590507 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 06:59:18.590871 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":279,"threads_started":1}
I20260927 06:59:18.593196 20714 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34877
I20260927 06:59:18.593490 20714 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:18.594310 20714 heartbeater.cc:511] Master 127.0.97.62:34877 requested a full tablet report, sending...
I20260927 06:59:18.595633 20267 ts_manager.cc:194] Registered new tserver with Master: 243bd786801848c6b694b0c593101d08 (127.0.97.3:42199)
I20260927 06:59:18.596371 20267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:52929
I20260927 06:59:18.603889 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:59:18.614169 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.608035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.0.97.62:34877)
0927 06:59:18.608214 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:18.608218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:18.608657 (+ 439us) server_negotiation.cc:407] Connection header received
0927 06:59:18.608848 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:18.608853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:18.608926 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:18.609031 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:18.609544 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.610290 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.611725 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.612117 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.612256 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.613397 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:18.613402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:18.613405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:18.613422 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:18.613451 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:18.613463 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:18.613562 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:18.613723 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 06:59:18.613809 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 06:59:18.617810 20267 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:59:18.624514 20267 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:59:18.645020 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.635688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42199 (local address 127.0.0.1:40794)
0927 06:59:18.636625 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:18.636679 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:18.636886 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:18.638125 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:18.638130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:18.638155 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:18.638507 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.638540 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.640362 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.640373 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.642086 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.642109 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.644081 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.644089 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.644307 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.644353 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 06:59:18.644474 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 06:59:18.644665 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":399,"threads_started":1}
I20260927 06:59:18.645071 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.635908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40794 (local address 127.0.97.3:42199)
0927 06:59:18.637334 (+ 1426us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:18.637347 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:18.637390 (+ 43us) server_negotiation.cc:407] Connection header received
0927 06:59:18.637513 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:18.637524 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:18.637800 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:18.638086 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:18.638751 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.640219 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.642379 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.643858 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.644006 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.644299 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:18.644555 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 06:59:18.644731 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1221,"thread_start_us":1061,"threads_started":1}
I20260927 06:59:18.647671 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.636913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40107 (local address 127.0.0.1:56946)
0927 06:59:18.638880 (+ 1967us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:18.638898 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:18.639015 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:18.639974 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:18.639982 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:18.640014 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:18.640526 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.640539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.642804 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.642808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.644663 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.644676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.646869 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.646876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.647052 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.647083 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 06:59:18.647207 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 06:59:18.647346 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1800,"thread_start_us":1662,"threads_started":1}
I20260927 06:59:18.648022 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.637366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.0.97.1:40107)
0927 06:59:18.637897 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:18.637903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:18.639145 (+ 1242us) server_negotiation.cc:407] Connection header received
0927 06:59:18.639264 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:18.639271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:18.639495 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:18.639676 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:18.640750 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.642629 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.644877 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.646340 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.646478 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.646715 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:18.647496 (+ 781us) server_negotiation.cc:299] Negotiation successful
0927 06:59:18.647663 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":284,"threads_started":1}
I20260927 06:59:18.649485 20649 tablet_service.cc:1467] Processing CreateTablet for tablet 7a65858f964346b08db9e69e09ee2d63 (DEFAULT_TABLE table=test-table [id=675058b734de4d65a37376bbbada467a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:18.650894 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.639562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36141 (local address 127.0.0.1:42952)
0927 06:59:18.642485 (+ 2923us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:18.642502 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:18.642682 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:18.643622 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:18.643626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:18.643654 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:18.644046 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.644062 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.645949 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.645954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.647903 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.647917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.650168 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.650175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.650336 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.650360 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 06:59:18.650468 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 06:59:18.650591 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2808,"thread_start_us":2642,"threads_started":1}
I20260927 06:59:18.651042 20649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a65858f964346b08db9e69e09ee2d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:18.651067 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.639269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.0.97.2:36141)
0927 06:59:18.640749 (+ 1480us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:18.640757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:18.642587 (+ 1830us) server_negotiation.cc:407] Connection header received
0927 06:59:18.642955 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:18.642963 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:18.643169 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:18.643313 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:18.644262 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.645807 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.648127 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.649671 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.649753 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.649941 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:18.650526 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 06:59:18.650683 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1319,"thread_start_us":1200,"threads_started":1}
I20260927 06:59:18.651518 20388 tablet_service.cc:1467] Processing CreateTablet for tablet 7a65858f964346b08db9e69e09ee2d63 (DEFAULT_TABLE table=test-table [id=675058b734de4d65a37376bbbada467a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:18.652851 20388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a65858f964346b08db9e69e09ee2d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:18.654302 20519 tablet_service.cc:1467] Processing CreateTablet for tablet 7a65858f964346b08db9e69e09ee2d63 (DEFAULT_TABLE table=test-table [id=675058b734de4d65a37376bbbada467a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:18.655282 20519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a65858f964346b08db9e69e09ee2d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:18.666359 20732 tablet_bootstrap.cc:492] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc: Bootstrap starting.
I20260927 06:59:18.669317 20732 tablet_bootstrap.cc:654] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:18.670200 20734 tablet_bootstrap.cc:492] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db: Bootstrap starting.
I20260927 06:59:18.670650 20732 log.cc:824] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:18.671793 20733 tablet_bootstrap.cc:492] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08: Bootstrap starting.
I20260927 06:59:18.673722 20734 tablet_bootstrap.cc:654] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:18.674268 20732 tablet_bootstrap.cc:492] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc: No bootstrap required, opened a new log
I20260927 06:59:18.674574 20732 ts_tablet_manager.cc:1397] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 06:59:18.674970 20734 log.cc:824] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:18.675371 20733 tablet_bootstrap.cc:654] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:18.676256 20733 log.cc:824] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:18.679025 20734 tablet_bootstrap.cc:492] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db: No bootstrap required, opened a new log
I20260927 06:59:18.679247 20733 tablet_bootstrap.cc:492] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08: No bootstrap required, opened a new log
I20260927 06:59:18.679328 20734 ts_tablet_manager.cc:1397] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 06:59:18.679493 20733 ts_tablet_manager.cc:1397] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260927 06:59:18.685567 20732 raft_consensus.cc:348] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243bd786801848c6b694b0c593101d08" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } } peers { permanent_uuid: "de006593ff774524b888b71d481056db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40107 } } peers { permanent_uuid: "fbfce91c028c48fdbf430f37f7734ebc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36141 } }
I20260927 06:59:18.686043 20732 raft_consensus.cc:374] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:18.686177 20732 raft_consensus.cc:729] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbfce91c028c48fdbf430f37f7734ebc, State: Initialized, Role: FOLLOWER
I20260927 06:59:18.686887 20732 consensus_queue.cc:260] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243bd786801848c6b694b0c593101d08" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } } peers { permanent_uuid: "de006593ff774524b888b71d481056db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40107 } } peers { permanent_uuid: "fbfce91c028c48fdbf430f37f7734ebc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36141 } }
I20260927 06:59:18.687227 20734 raft_consensus.cc:348] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243bd786801848c6b694b0c593101d08" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } } peers { permanent_uuid: "de006593ff774524b888b71d481056db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40107 } } peers { permanent_uuid: "fbfce91c028c48fdbf430f37f7734ebc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36141 } }
I20260927 06:59:18.687599 20734 raft_consensus.cc:374] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:18.687712 20734 raft_consensus.cc:729] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de006593ff774524b888b71d481056db, State: Initialized, Role: FOLLOWER
I20260927 06:59:18.688278 20734 consensus_queue.cc:260] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243bd786801848c6b694b0c593101d08" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } } peers { permanent_uuid: "de006593ff774524b888b71d481056db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40107 } } peers { permanent_uuid: "fbfce91c028c48fdbf430f37f7734ebc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36141 } }
I20260927 06:59:18.690774 20733 raft_consensus.cc:348] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243bd786801848c6b694b0c593101d08" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } } peers { permanent_uuid: "de006593ff774524b888b71d481056db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40107 } } peers { permanent_uuid: "fbfce91c028c48fdbf430f37f7734ebc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36141 } }
I20260927 06:59:18.691323 20733 raft_consensus.cc:374] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:18.691448 20733 raft_consensus.cc:729] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 243bd786801848c6b694b0c593101d08, State: Initialized, Role: FOLLOWER
I20260927 06:59:18.692104 20733 consensus_queue.cc:260] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243bd786801848c6b694b0c593101d08" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } } peers { permanent_uuid: "de006593ff774524b888b71d481056db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40107 } } peers { permanent_uuid: "fbfce91c028c48fdbf430f37f7734ebc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36141 } }
I20260927 06:59:18.694377 20584 heartbeater.cc:503] Master 127.0.97.62:34877 was elected leader, sending a full tablet report...
I20260927 06:59:18.695591 20732 ts_tablet_manager.cc:1428] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc: Time spent starting tablet: real 0.021s user 0.006s sys 0.013s
I20260927 06:59:18.696604 20453 heartbeater.cc:503] Master 127.0.97.62:34877 was elected leader, sending a full tablet report...
I20260927 06:59:18.696979 20714 heartbeater.cc:503] Master 127.0.97.62:34877 was elected leader, sending a full tablet report...
I20260927 06:59:18.697168 20734 ts_tablet_manager.cc:1428] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db: Time spent starting tablet: real 0.018s user 0.007s sys 0.011s
I20260927 06:59:18.700675 20733 ts_tablet_manager.cc:1428] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08: Time spent starting tablet: real 0.021s user 0.012s sys 0.007s
W20260927 06:59:18.704638 20454 tablet.cc:2369] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:59:18.830087 20715 tablet.cc:2369] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:59:18.861582 20738 raft_consensus.cc:482] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:59:18.861960 20738 raft_consensus.cc:504] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243bd786801848c6b694b0c593101d08" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } } peers { permanent_uuid: "de006593ff774524b888b71d481056db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40107 } } peers { permanent_uuid: "fbfce91c028c48fdbf430f37f7734ebc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36141 } }
I20260927 06:59:18.863909 20738 leader_election.cc:288] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 243bd786801848c6b694b0c593101d08 (127.0.97.3:42199), de006593ff774524b888b71d481056db (127.0.97.1:40107)
I20260927 06:59:18.870741 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.865101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59243 (local address 127.0.97.1:40107)
0927 06:59:18.865351 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:18.865357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:18.865624 (+ 267us) server_negotiation.cc:407] Connection header received
0927 06:59:18.865864 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:18.865869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:18.865948 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:18.866092 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:18.866620 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.867318 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.869043 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.869783 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.869934 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.870055 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:18.870365 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 06:59:18.870482 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260927 06:59:18.870898 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.864924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40107 (local address 127.0.97.2:59243)
0927 06:59:18.865506 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:18.865520 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:18.865714 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:18.866105 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:18.866109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:18.866132 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:18.866464 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.866473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.867461 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.867467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.868853 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.868864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.869970 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.869976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.870181 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.870208 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 06:59:18.870340 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 06:59:18.870463 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":3,"spinlock_wait_cycles":17024,"thread_start_us":313,"threads_started":1}
I20260927 06:59:18.870904 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.864372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44867 (local address 127.0.97.3:42199)
0927 06:59:18.864570 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:18.864575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:18.864971 (+ 396us) server_negotiation.cc:407] Connection header received
0927 06:59:18.865290 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:18.865297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:18.865382 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:18.865464 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:18.866261 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.867046 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.869051 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.869866 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.870051 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.870159 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:18.870484 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 06:59:18.870630 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 06:59:18.871062 20741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.864092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42199 (local address 127.0.97.2:44867)
0927 06:59:18.864834 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:18.864856 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:18.865015 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:18.865648 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:18.865652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:18.865707 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:18.866084 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.866098 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.867214 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.867223 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.868824 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.868836 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.870097 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.870104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.870310 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.870333 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 06:59:18.870457 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 06:59:18.870590 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":2,"spinlock_wait_cycles":14080,"thread_start_us":358,"threads_started":1}
I20260927 06:59:18.872133 20669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a65858f964346b08db9e69e09ee2d63" candidate_uuid: "fbfce91c028c48fdbf430f37f7734ebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "243bd786801848c6b694b0c593101d08" is_pre_election: true
I20260927 06:59:18.872169 20408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a65858f964346b08db9e69e09ee2d63" candidate_uuid: "fbfce91c028c48fdbf430f37f7734ebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de006593ff774524b888b71d481056db" is_pre_election: true
I20260927 06:59:18.872658 20669 raft_consensus.cc:2454] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbfce91c028c48fdbf430f37f7734ebc in term 0.
I20260927 06:59:18.872754 20408 raft_consensus.cc:2454] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbfce91c028c48fdbf430f37f7734ebc in term 0.
I20260927 06:59:18.873415 20472 leader_election.cc:302] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 243bd786801848c6b694b0c593101d08, fbfce91c028c48fdbf430f37f7734ebc; no voters:
I20260927 06:59:18.873749 20738 raft_consensus.cc:2781] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:59:18.873911 20738 raft_consensus.cc:482] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:59:18.874080 20738 raft_consensus.cc:3037] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:18.877986 20738 raft_consensus.cc:504] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243bd786801848c6b694b0c593101d08" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } } peers { permanent_uuid: "de006593ff774524b888b71d481056db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40107 } } peers { permanent_uuid: "fbfce91c028c48fdbf430f37f7734ebc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36141 } }
I20260927 06:59:18.878926 20738 leader_election.cc:288] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [CANDIDATE]: Term 1 election: Requested vote from peers 243bd786801848c6b694b0c593101d08 (127.0.97.3:42199), de006593ff774524b888b71d481056db (127.0.97.1:40107)
I20260927 06:59:18.879459 20669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a65858f964346b08db9e69e09ee2d63" candidate_uuid: "fbfce91c028c48fdbf430f37f7734ebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "243bd786801848c6b694b0c593101d08"
I20260927 06:59:18.879459 20408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a65858f964346b08db9e69e09ee2d63" candidate_uuid: "fbfce91c028c48fdbf430f37f7734ebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de006593ff774524b888b71d481056db"
I20260927 06:59:18.879694 20408 raft_consensus.cc:3037] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:18.879690 20669 raft_consensus.cc:3037] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:18.883296 20408 raft_consensus.cc:2454] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbfce91c028c48fdbf430f37f7734ebc in term 1.
I20260927 06:59:18.883296 20669 raft_consensus.cc:2454] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbfce91c028c48fdbf430f37f7734ebc in term 1.
I20260927 06:59:18.883687 20472 leader_election.cc:302] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 243bd786801848c6b694b0c593101d08, fbfce91c028c48fdbf430f37f7734ebc; no voters:
I20260927 06:59:18.884020 20738 raft_consensus.cc:2781] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:18.884591 20738 raft_consensus.cc:686] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [term 1 LEADER]: Becoming Leader. State: Replica: fbfce91c028c48fdbf430f37f7734ebc, State: Running, Role: LEADER
I20260927 06:59:18.885098 20738 consensus_queue.cc:237] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243bd786801848c6b694b0c593101d08" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } } peers { permanent_uuid: "de006593ff774524b888b71d481056db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40107 } } peers { permanent_uuid: "fbfce91c028c48fdbf430f37f7734ebc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36141 } }
I20260927 06:59:18.889966 20265 catalog_manager.cc:5495] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc reported cstate change: term changed from 0 to 1, leader changed from <none> to fbfce91c028c48fdbf430f37f7734ebc (127.0.97.2). New cstate: current_term: 1 leader_uuid: "fbfce91c028c48fdbf430f37f7734ebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243bd786801848c6b694b0c593101d08" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de006593ff774524b888b71d481056db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbfce91c028c48fdbf430f37f7734ebc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36141 } health_report { overall_health: HEALTHY } } }
W20260927 06:59:18.893234 20585 tablet.cc:2369] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:59:18.941118 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.936169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.0.97.2:36141)
0927 06:59:18.936367 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:18.936371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:18.936391 (+ 20us) server_negotiation.cc:407] Connection header received
0927 06:59:18.936485 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:18.936489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:18.936560 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:18.936675 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:18.937021 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.937671 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.939252 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.939623 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.939714 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.940193 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:18.940692 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:18.940797 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:18.940880 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 06:59:18.940938 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 06:59:18.948913 20669 raft_consensus.cc:1261] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08 [term 1 FOLLOWER]: Refusing update from remote peer fbfce91c028c48fdbf430f37f7734ebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:18.948915 20408 raft_consensus.cc:1261] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db [term 1 FOLLOWER]: Refusing update from remote peer fbfce91c028c48fdbf430f37f7734ebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:18.949838 20743 consensus_queue.cc:1035] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "243bd786801848c6b694b0c593101d08" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:18.950235 20738 consensus_queue.cc:1035] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "de006593ff774524b888b71d481056db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:18.963183 20747 mvcc.cc:205] Tried to move back new op lower bound from 7333856702244913152 to 7333856701997924352. Current Snapshot: MvccSnapshot[applied={T|T < 7333856702244913152}]
I20260927 06:59:18.973060 20265 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:18.973304 20265 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:18.976563 20265 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3378e67ace7f4e0694db3aac49fed038: Sending DeleteTablet for 3 replicas of tablet 7a65858f964346b08db9e69e09ee2d63
I20260927 06:59:18.977562 20649 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a65858f964346b08db9e69e09ee2d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:18 UTC) from {username='slave'} at 127.0.0.1:40794
I20260927 06:59:18.977717 20388 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a65858f964346b08db9e69e09ee2d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:18 UTC) from {username='slave'} at 127.0.0.1:56946
I20260927 06:59:18.977881 20519 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a65858f964346b08db9e69e09ee2d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:18 UTC) from {username='slave'} at 127.0.0.1:42952
I20260927 06:59:18.978991 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 20326
I20260927 06:59:18.985232 20757 tablet_replica.cc:330] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db: stopping tablet replica
I20260927 06:59:18.990108 20756 tablet_replica.cc:330] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08: stopping tablet replica
I20260927 06:59:18.990790 20758 tablet_replica.cc:330] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc: stopping tablet replica
I20260927 06:59:18.990979 20757 raft_consensus.cc:2229] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:18.991223 20758 raft_consensus.cc:2229] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:59:18.991369 20756 raft_consensus.cc:2229] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:18.991590 20757 raft_consensus.cc:2258] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:18.991955 20758 raft_consensus.cc:2258] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:18.992012 20756 raft_consensus.cc:2258] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:18.993999 20758 ts_tablet_manager.cc:1905] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:18.994033 20757 ts_tablet_manager.cc:1905] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:18.994029 20756 ts_tablet_manager.cc:1905] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:19.000844 20756 ts_tablet_manager.cc:1918] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:19.001052 20756 log.cc:1196] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-2/wal/wals/7a65858f964346b08db9e69e09ee2d63
I20260927 06:59:19.001196 20758 ts_tablet_manager.cc:1918] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:19.001190 20757 ts_tablet_manager.cc:1918] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:19.001348 20757 log.cc:1196] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-0/wal/wals/7a65858f964346b08db9e69e09ee2d63
I20260927 06:59:19.001470 20758 log.cc:1196] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc: Deleting WAL directory at /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492047466241-388-0/minicluster-data/ts-1/wal/wals/7a65858f964346b08db9e69e09ee2d63
I20260927 06:59:19.001638 20756 ts_tablet_manager.cc:1939] T 7a65858f964346b08db9e69e09ee2d63 P 243bd786801848c6b694b0c593101d08: Deleting consensus metadata
I20260927 06:59:19.001902 20757 ts_tablet_manager.cc:1939] T 7a65858f964346b08db9e69e09ee2d63 P de006593ff774524b888b71d481056db: Deleting consensus metadata
I20260927 06:59:19.002120 20758 ts_tablet_manager.cc:1939] T 7a65858f964346b08db9e69e09ee2d63 P fbfce91c028c48fdbf430f37f7734ebc: Deleting consensus metadata
I20260927 06:59:19.003513 20251 catalog_manager.cc:4841] TS 243bd786801848c6b694b0c593101d08 (127.0.97.3:42199): tablet 7a65858f964346b08db9e69e09ee2d63 (table test-table [id=675058b734de4d65a37376bbbada467a]) successfully deleted
I20260927 06:59:19.003868 20251 catalog_manager.cc:4841] TS fbfce91c028c48fdbf430f37f7734ebc (127.0.97.2:36141): tablet 7a65858f964346b08db9e69e09ee2d63 (table test-table [id=675058b734de4d65a37376bbbada467a]) successfully deleted
I20260927 06:59:19.004112 20251 catalog_manager.cc:4841] TS de006593ff774524b888b71d481056db (127.0.97.1:40107): tablet 7a65858f964346b08db9e69e09ee2d63 (table test-table [id=675058b734de4d65a37376bbbada467a]) successfully deleted
I20260927 06:59:19.005890 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.980239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56952 (local address 127.0.97.1:40107)
0927 06:59:18.980422 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:18.980429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:18.991092 (+ 10663us) server_negotiation.cc:407] Connection header received
0927 06:59:18.991368 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:18.991375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:18.991507 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:18.991591 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:18.992260 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.993387 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.995152 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.995852 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.995951 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:19.004423 (+ 8472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:19.004434 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:19.004450 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:19.005131 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:19.005189 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:19.005220 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:19.005347 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:19.005441 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 06:59:19.005546 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 06:59:19.006599 20448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:19.095788 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 20326
I20260927 06:59:19.119880 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 20457
I20260927 06:59:19.127175 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:19.120696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.0.97.2:36141)
0927 06:59:19.120878 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:19.120886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:19.120907 (+ 21us) server_negotiation.cc:407] Connection header received
0927 06:59:19.121128 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:19.121133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:19.121255 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:19.121373 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:19.121815 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:19.122623 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:19.123901 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:19.124319 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:19.124470 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:19.125614 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:19.125622 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:19.125635 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:19.126354 (+ 719us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:19.126407 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:19.126426 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:19.126549 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:19.126700 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 06:59:19.126798 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 06:59:19.127903 20579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:19.236781 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 20457
I20260927 06:59:19.255103 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 20587
I20260927 06:59:19.261183 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:19.255847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40806 (local address 127.0.97.3:42199)
0927 06:59:19.256011 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:19.256016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:19.256034 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:59:19.256175 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:19.256179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:19.256271 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:19.256357 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:19.256689 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:19.257361 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:19.258593 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:19.258979 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:19.259069 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:19.260015 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:19.260023 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:19.260036 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:19.260587 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:19.260641 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:19.260660 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:19.260745 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:19.260866 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 06:59:19.260948 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 06:59:19.261844 20709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:19.350304 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 20587
I20260927 06:59:19.367136 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 20236
I20260927 06:59:19.368228 20297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:19.476608 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 20236
2026-09-27T06:59:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2811 ms)
[----------] 6 tests from EncryptionPolicyTest (21267 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:19 dist-test-slave-f892 krb5kdc[20766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:19 dist-test-slave-f892 krb5kdc[20766](info): set up 2 sockets
Sep 27 06:59:19 dist-test-slave-f892 krb5kdc[20766](info): commencing operation
krb5kdc: starting...
W20260927 06:59:20.578464 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:20 dist-test-slave-f892 krb5kdc[20766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 06:59:20.821743 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34159
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:21.139603 20779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:21.139930 20779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:21.140098 20779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:21.147279 20779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:21.147431 20779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:21.147535 20779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:21.147639 20779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:21.147719 20779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:21.157279 20779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34159
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:21.158977 20779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:21.161211 20779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:21.176481 20784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:21.177631 20787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:21.177897 20779 server_base.cc:1034] running on GCE node
W20260927 06:59:21.178700 20785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:21.179720 20779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:21.181150 20779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:21.182385 20779 hybrid_clock.cc:648] HybridClock initialized: now 1790492361182333 us; error 45 us; skew 500 ppm
I20260927 06:59:21.182749 20779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:21 dist-test-slave-f892 krb5kdc[20766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:21.186504 20779 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 06:59:21.188161 20779 webserver.cc:460] Webserver started at http://127.0.0.1:33531/ using document root <none> and password file <none>
I20260927 06:59:21.188941 20779 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:21.189072 20779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:21.189383 20779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:21.193305 20779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "bb31c7b506e34c0ca634b6e71dec1494"
format_stamp: "Formatted at 2026-09-27 06:59:21 on dist-test-slave-f892"
server_key: "2cd71484e7824f615ef4cfa788ef9d8b"
server_key_iv: "810dc12f75cc64b166455c0dbc453142"
server_key_version: "encryptionkey@0"
I20260927 06:59:21.194332 20779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "bb31c7b506e34c0ca634b6e71dec1494"
format_stamp: "Formatted at 2026-09-27 06:59:21 on dist-test-slave-f892"
server_key: "2cd71484e7824f615ef4cfa788ef9d8b"
server_key_iv: "810dc12f75cc64b166455c0dbc453142"
server_key_version: "encryptionkey@0"
I20260927 06:59:21.201443 20779 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260927 06:59:21.206024 20794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:21.207492 20779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 06:59:21.207687 20779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "bb31c7b506e34c0ca634b6e71dec1494"
format_stamp: "Formatted at 2026-09-27 06:59:21 on dist-test-slave-f892"
server_key: "2cd71484e7824f615ef4cfa788ef9d8b"
server_key_iv: "810dc12f75cc64b166455c0dbc453142"
server_key_version: "encryptionkey@0"
I20260927 06:59:21.207885 20779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:21.238703 20779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:21.243211 20779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:21.243511 20779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:21.262648 20779 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34159
I20260927 06:59:21.262691 20845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34159 every 8 connection(s)
I20260927 06:59:21.264006 20779 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:21.264636 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 20779
I20260927 06:59:21.264933 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:21.265779 388 external_mini_cluster.cc:1428] Setting key 06fd3eaecda8654b74dee58da2c5b7a1
I20260927 06:59:21.268497 20846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 06:59:21 dist-test-slave-f892 krb5kdc[20766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 06:59:21.282058 20846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494: Bootstrap starting.
I20260927 06:59:21.286264 20846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:21.287477 20846 log.cc:824] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:21.290787 20846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494: No bootstrap required, opened a new log
I20260927 06:59:21.291133 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:21.267955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45445 (local address 127.0.0.1:34159)
0927 06:59:21.268774 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:21.268786 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:21.268829 (+ 43us) server_negotiation.cc:407] Connection header received
0927 06:59:21.270845 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:21.270866 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:21.271247 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:21.271578 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:21.273369 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:21.274817 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:21.276619 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:21.277267 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:21.277392 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:21.282152 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:21.282160 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:21.282174 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:21.282238 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:21.286289 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:21.287430 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:21.287438 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:21.287446 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:21.287672 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:21.288233 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:21.288236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:21.288241 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:21.288770 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:21.289102 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:21.289473 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 06:59:21.289795 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":365,"threads_started":1}
I20260927 06:59:21.295529 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:21.302778 20846 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb31c7b506e34c0ca634b6e71dec1494" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34159 } }
I20260927 06:59:21.303267 20846 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:21.303387 20846 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31c7b506e34c0ca634b6e71dec1494, State: Initialized, Role: FOLLOWER
I20260927 06:59:21.304128 20846 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb31c7b506e34c0ca634b6e71dec1494" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34159 } }
I20260927 06:59:21.304433 20846 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:21.304606 20846 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:21.304831 20846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:21.308277 20846 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb31c7b506e34c0ca634b6e71dec1494" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34159 } }
I20260927 06:59:21.308897 20846 leader_election.cc:302] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb31c7b506e34c0ca634b6e71dec1494; no voters:
I20260927 06:59:21.309585 20846 leader_election.cc:288] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:21.309695 20856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:21.312394 20846 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:21.314347 20856 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494 [term 1 LEADER]: Becoming Leader. State: Replica: bb31c7b506e34c0ca634b6e71dec1494, State: Running, Role: LEADER
I20260927 06:59:21.315141 20856 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb31c7b506e34c0ca634b6e71dec1494" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34159 } }
I20260927 06:59:21.318552 20858 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb31c7b506e34c0ca634b6e71dec1494. Latest consensus state: current_term: 1 leader_uuid: "bb31c7b506e34c0ca634b6e71dec1494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb31c7b506e34c0ca634b6e71dec1494" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34159 } } }
I20260927 06:59:21.319504 20858 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:21.319262 20857 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb31c7b506e34c0ca634b6e71dec1494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb31c7b506e34c0ca634b6e71dec1494" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34159 } } }
I20260927 06:59:21.319680 20857 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:21.320879 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:21.301449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37861 (local address 127.0.0.1:34159)
0927 06:59:21.301669 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:21.301677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:21.302309 (+ 632us) server_negotiation.cc:407] Connection header received
0927 06:59:21.302533 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:21.302560 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:21.302657 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:21.302774 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:21.304867 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:21.305672 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:21.307723 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:21.308143 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:21.308232 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:21.311530 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:21.311538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:21.311543 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:21.311586 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:21.315091 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:21.315978 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:21.315983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:21.315988 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:21.316135 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:21.316696 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:21.316701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:21.316704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:21.317077 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:21.319020 (+ 1943us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:21.320261 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0927 06:59:21.320461 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 06:59:21.321162 20861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:21.326112 20861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:21.331004 20873 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 06:59:21.335109 20861 catalog_manager.cc:1347] Generated new cluster ID: 70df19525fd74ed48813ec8a12be9427
I20260927 06:59:21.335253 20861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:21.362769 20861 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:21.364676 20861 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:21.373354 20861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb31c7b506e34c0ca634b6e71dec1494: Generated new TSK 0
I20260927 06:59:21.374466 20861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:21.384188 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 20779
I20260927 06:59:21.385732 20841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:21.478513 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 20779
W20260927 06:59:21.499641 20764 connection.cc:535] client connection to 127.0.0.1:34159 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34159): Connection reset by peer (error 104)
2026-09-27T06:59:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2000 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:21 dist-test-slave-f892 krb5kdc[20878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:21 dist-test-slave-f892 krb5kdc[20878](info): set up 2 sockets
Sep 27 06:59:21 dist-test-slave-f892 krb5kdc[20878](info): commencing operation
krb5kdc: starting...
W20260927 06:59:22.578370 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:22 dist-test-slave-f892 krb5kdc[20878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 06:59:22.819375 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:23.141800 20891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:23.142150 20891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:23.142263 20891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:23.149382 20891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:23.149482 20891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:23.149529 20891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:23.149597 20891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:23.149636 20891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:23.159188 20891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45551
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:23.160346 20891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:23.161602 20891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:23.173401 20896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:23.173348 20897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:23.173449 20899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:23.174225 20891 server_base.cc:1034] running on GCE node
I20260927 06:59:23.174902 20891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:23.176121 20891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:23.177322 20891 hybrid_clock.cc:648] HybridClock initialized: now 1790492363177284 us; error 39 us; skew 500 ppm
I20260927 06:59:23.177623 20891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:23 dist-test-slave-f892 krb5kdc[20878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:23.180630 20891 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 06:59:23.182303 20891 webserver.cc:460] Webserver started at http://127.0.0.1:46155/ using document root <none> and password file <none>
I20260927 06:59:23.182932 20891 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:23.183099 20891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:23.183391 20891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:23.186196 20891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "1db7585cac5145eaa1ff0e85eaf41a89"
format_stamp: "Formatted at 2026-09-27 06:59:23 on dist-test-slave-f892"
server_key: "44b0116badccb280b6f3a8a81a51374d"
server_key_iv: "7c6194802f76e6b7eab0fdd801a5a345"
server_key_version: "encryptionkey@0"
I20260927 06:59:23.186933 20891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "1db7585cac5145eaa1ff0e85eaf41a89"
format_stamp: "Formatted at 2026-09-27 06:59:23 on dist-test-slave-f892"
server_key: "44b0116badccb280b6f3a8a81a51374d"
server_key_iv: "7c6194802f76e6b7eab0fdd801a5a345"
server_key_version: "encryptionkey@0"
I20260927 06:59:23.191915 20891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 06:59:23.195247 20906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:23.196449 20891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 06:59:23.196641 20891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "1db7585cac5145eaa1ff0e85eaf41a89"
format_stamp: "Formatted at 2026-09-27 06:59:23 on dist-test-slave-f892"
server_key: "44b0116badccb280b6f3a8a81a51374d"
server_key_iv: "7c6194802f76e6b7eab0fdd801a5a345"
server_key_version: "encryptionkey@0"
I20260927 06:59:23.196834 20891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:23.228546 20891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:23.232955 20891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:23.233246 20891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:23.250838 20891 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45551
I20260927 06:59:23.250876 20957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45551 every 8 connection(s)
I20260927 06:59:23.252202 20891 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:23.252420 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 20891
I20260927 06:59:23.252609 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:23.253268 388 external_mini_cluster.cc:1428] Setting key 6e9a3b4187e698aa9cd98282307b1d67
I20260927 06:59:23.256778 20958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:23.268286 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:23.255116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42119 (local address 127.0.0.1:45551)
0927 06:59:23.255889 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:23.255903 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:23.255950 (+ 47us) server_negotiation.cc:407] Connection header received
0927 06:59:23.258243 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:23.258269 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:23.258744 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:23.259129 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:23.259633 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:23.261614 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:23.263175 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:23.264139 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:23.264319 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:23.265408 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:23.265418 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:23.265437 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:23.266198 (+ 761us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:23.266291 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:23.266321 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:23.266500 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:23.266773 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 06:59:23.266935 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":302,"threads_started":1}
I20260927 06:59:23.270298 20958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89: Bootstrap starting.
I20260927 06:59:23.271850 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:23.273391 20958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:23.274446 20958 log.cc:824] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:23.277307 20958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89: No bootstrap required, opened a new log
I20260927 06:59:23.282644 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:23.276097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38961 (local address 127.0.0.1:45551)
0927 06:59:23.276244 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:23.276249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:23.276607 (+ 358us) server_negotiation.cc:407] Connection header received
0927 06:59:23.276730 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:23.276735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:23.276800 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:23.276905 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:23.277410 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:23.278563 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:23.279866 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:23.280576 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:23.280717 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:23.281800 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:23.281806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:23.281811 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:23.281830 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:23.281881 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:23.281905 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:23.282037 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:23.282256 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 06:59:23.282309 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 06:59:23.285768 20958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7585cac5145eaa1ff0e85eaf41a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45551 } }
I20260927 06:59:23.286164 20958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:23.286244 20958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db7585cac5145eaa1ff0e85eaf41a89, State: Initialized, Role: FOLLOWER
I20260927 06:59:23.286842 20958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7585cac5145eaa1ff0e85eaf41a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45551 } }
I20260927 06:59:23.287097 20958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:23.287245 20958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:23.287423 20958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:23.289494 20958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7585cac5145eaa1ff0e85eaf41a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45551 } }
I20260927 06:59:23.289971 20958 leader_election.cc:302] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1db7585cac5145eaa1ff0e85eaf41a89; no voters:
I20260927 06:59:23.290587 20958 leader_election.cc:288] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:23.290732 20968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:23.292948 20958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:23.294499 20968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89 [term 1 LEADER]: Becoming Leader. State: Replica: 1db7585cac5145eaa1ff0e85eaf41a89, State: Running, Role: LEADER
I20260927 06:59:23.295151 20968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7585cac5145eaa1ff0e85eaf41a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45551 } }
I20260927 06:59:23.298611 20969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1db7585cac5145eaa1ff0e85eaf41a89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7585cac5145eaa1ff0e85eaf41a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45551 } } }
I20260927 06:59:23.299096 20969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:23.298658 20970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1db7585cac5145eaa1ff0e85eaf41a89. Latest consensus state: current_term: 1 leader_uuid: "1db7585cac5145eaa1ff0e85eaf41a89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7585cac5145eaa1ff0e85eaf41a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45551 } } }
I20260927 06:59:23.304597 20970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:23.310973 20979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:23.316232 20979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:23.325497 20979 catalog_manager.cc:1347] Generated new cluster ID: ef75326b21fc4952a8008e191d313ed6
I20260927 06:59:23.325644 20979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:23.349578 20979 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:23.350891 20979 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:23.361821 20979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1db7585cac5145eaa1ff0e85eaf41a89: Generated new TSK 0
I20260927 06:59:23.362697 20979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:23.373154 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 20891
W20260927 06:59:23.374387 20953 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42119
W20260927 06:59:23.375101 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 20891: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:59:23.375281 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 20891
W20260927 06:59:23.392092 20876 connection.cc:535] client connection to 127.0.0.1:45551 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45551): Connection reset by peer (error 104)
2026-09-27T06:59:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1892 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:23 dist-test-slave-f892 krb5kdc[20989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:23 dist-test-slave-f892 krb5kdc[20989](info): set up 2 sockets
Sep 27 06:59:23 dist-test-slave-f892 krb5kdc[20989](info): commencing operation
krb5kdc: starting...
W20260927 06:59:24.466320 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:24 dist-test-slave-f892 krb5kdc[20989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 06:59:24.695250 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:25.011615 21002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:25.011883 21002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:25.011972 21002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:25.019030 21002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:25.019150 21002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:25.019194 21002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:25.019255 21002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:25.019302 21002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:25.028934 21002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46437
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:25.030088 21002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:25.031316 21002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:25.042475 21007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:25.042531 21010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:25.042441 21008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:25.043454 21002 server_base.cc:1034] running on GCE node
I20260927 06:59:25.044183 21002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:25.045462 21002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:25.046686 21002 hybrid_clock.cc:648] HybridClock initialized: now 1790492365046644 us; error 57 us; skew 500 ppm
I20260927 06:59:25.047024 21002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:25 dist-test-slave-f892 krb5kdc[20989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:25.052846 21002 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 06:59:25.054278 21002 webserver.cc:460] Webserver started at http://127.0.0.1:43691/ using document root <none> and password file <none>
I20260927 06:59:25.054925 21002 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:25.055073 21002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:25.055410 21002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:25.058216 21002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "0ce79be2290a40cf99f7531d94691546"
format_stamp: "Formatted at 2026-09-27 06:59:25 on dist-test-slave-f892"
server_key: "2f35aaa26a93576a292c093c135dfde9"
server_key_iv: "c4245a632cb1955dd64b8d82745bd4c3"
server_key_version: "encryptionkey@0"
I20260927 06:59:25.058926 21002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "0ce79be2290a40cf99f7531d94691546"
format_stamp: "Formatted at 2026-09-27 06:59:25 on dist-test-slave-f892"
server_key: "2f35aaa26a93576a292c093c135dfde9"
server_key_iv: "c4245a632cb1955dd64b8d82745bd4c3"
server_key_version: "encryptionkey@0"
I20260927 06:59:25.063630 21002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 06:59:25.066751 21017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:25.067950 21002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 06:59:25.068122 21002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "0ce79be2290a40cf99f7531d94691546"
format_stamp: "Formatted at 2026-09-27 06:59:25 on dist-test-slave-f892"
server_key: "2f35aaa26a93576a292c093c135dfde9"
server_key_iv: "c4245a632cb1955dd64b8d82745bd4c3"
server_key_version: "encryptionkey@0"
I20260927 06:59:25.068326 21002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:25.073596 21002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:25.077194 21002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:25.077466 21002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:25.095471 21002 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46437
I20260927 06:59:25.095549 21068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46437 every 8 connection(s)
I20260927 06:59:25.096813 21002 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:25.100309 21069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:25.106202 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 21002
I20260927 06:59:25.106369 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:25.106789 388 external_mini_cluster.cc:1428] Setting key 051f808840b97d40030623163977d7c3
I20260927 06:59:25.110499 21069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546: Bootstrap starting.
I20260927 06:59:25.113914 21069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:25.114351 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:25.108448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56047 (local address 127.0.0.1:46437)
0927 06:59:25.109288 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:25.109298 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:25.109334 (+ 36us) server_negotiation.cc:407] Connection header received
0927 06:59:25.111143 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:25.111164 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:25.111458 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:25.111748 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:25.112799 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:25.112805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:25.112816 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:25.112830 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:25.112886 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:25.112904 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:25.113057 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:25.113223 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 06:59:25.113413 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":384,"threads_started":1}
I20260927 06:59:25.115234 21069 log.cc:824] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:25.118223 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:25.118342 21069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546: No bootstrap required, opened a new log
I20260927 06:59:25.125042 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:25.122411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43961 (local address 127.0.0.1:46437)
0927 06:59:25.122611 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:25.122616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:25.123044 (+ 428us) server_negotiation.cc:407] Connection header received
0927 06:59:25.123238 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:25.123243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:25.123313 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:25.123434 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:25.124367 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:25.124373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:25.124377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:25.124398 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:25.124449 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:25.124474 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:25.124631 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:25.124720 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 06:59:25.124777 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 06:59:25.126662 21069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce79be2290a40cf99f7531d94691546" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46437 } }
I20260927 06:59:25.127076 21069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:25.127185 21069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce79be2290a40cf99f7531d94691546, State: Initialized, Role: FOLLOWER
I20260927 06:59:25.127736 21069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce79be2290a40cf99f7531d94691546" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46437 } }
I20260927 06:59:25.127985 21069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:25.128127 21069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:25.128306 21069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:25.130800 21069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce79be2290a40cf99f7531d94691546" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46437 } }
I20260927 06:59:25.131290 21069 leader_election.cc:302] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ce79be2290a40cf99f7531d94691546; no voters:
I20260927 06:59:25.131856 21069 leader_election.cc:288] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:25.131994 21079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:25.132676 21079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546 [term 1 LEADER]: Becoming Leader. State: Replica: 0ce79be2290a40cf99f7531d94691546, State: Running, Role: LEADER
I20260927 06:59:25.133246 21079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce79be2290a40cf99f7531d94691546" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46437 } }
I20260927 06:59:25.133850 21069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:25.135684 21080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ce79be2290a40cf99f7531d94691546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce79be2290a40cf99f7531d94691546" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46437 } } }
I20260927 06:59:25.136018 21080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:25.138563 21085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:25.138425 21081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ce79be2290a40cf99f7531d94691546. Latest consensus state: current_term: 1 leader_uuid: "0ce79be2290a40cf99f7531d94691546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce79be2290a40cf99f7531d94691546" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46437 } } }
I20260927 06:59:25.138752 21081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:25.143235 21085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:25.153374 21085 catalog_manager.cc:1347] Generated new cluster ID: 1fccf197a32741659dfdd0c90ae97582
I20260927 06:59:25.153488 21085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:25.171420 21085 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:25.171864 21085 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:25.178923 21085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ce79be2290a40cf99f7531d94691546: Generated new TSK 0
I20260927 06:59:25.179751 21085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:25.183712 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 21002
W20260927 06:59:25.184727 21064 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56047
W20260927 06:59:25.185465 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 21002: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:59:25.185616 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 21002
2026-09-27T06:59:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1810 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:25 dist-test-slave-f892 krb5kdc[21100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:25 dist-test-slave-f892 krb5kdc[21100](info): set up 2 sockets
Sep 27 06:59:25 dist-test-slave-f892 krb5kdc[21100](info): commencing operation
krb5kdc: starting...
W20260927 06:59:26.282330 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:26 dist-test-slave-f892 krb5kdc[21100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 06:59:26.525288 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:26.840215 21113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:26.840482 21113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:26.840683 21113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:26.848524 21113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:26.848635 21113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:26.848747 21113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:26.848834 21113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:26.848897 21113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:26.858716 21113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:26.859977 21113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:26.861304 21113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:26.872875 21118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:26.872840 21121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:26.872932 21119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:26.873844 21113 server_base.cc:1034] running on GCE node
I20260927 06:59:26.874598 21113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:26.878319 21113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:26.879551 21113 hybrid_clock.cc:648] HybridClock initialized: now 1790492366879449 us; error 92 us; skew 500 ppm
I20260927 06:59:26.880019 21113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:26 dist-test-slave-f892 krb5kdc[21100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:26.884246 21113 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 06:59:26.885648 21113 webserver.cc:460] Webserver started at http://127.0.0.1:44985/ using document root <none> and password file <none>
I20260927 06:59:26.886397 21113 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:26.886554 21113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:26.886875 21113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:26.889766 21113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "bbf24fd8fa9d424fa2d4eea6c243d0d4"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-f892"
server_key: "9d6decd91205dbc99364261d8b43cc1e"
server_key_iv: "dfd73ecbb49f75e82be4b290d1a4db30"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.890588 21113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "bbf24fd8fa9d424fa2d4eea6c243d0d4"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-f892"
server_key: "9d6decd91205dbc99364261d8b43cc1e"
server_key_iv: "dfd73ecbb49f75e82be4b290d1a4db30"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.895488 21113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 06:59:26.898803 21128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.900089 21113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 06:59:26.900285 21113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "bbf24fd8fa9d424fa2d4eea6c243d0d4"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-f892"
server_key: "9d6decd91205dbc99364261d8b43cc1e"
server_key_iv: "dfd73ecbb49f75e82be4b290d1a4db30"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.900467 21113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:26.922490 21113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:26.926186 21113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:26.926446 21113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:26.944315 21113 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43781
I20260927 06:59:26.944357 21179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43781 every 8 connection(s)
I20260927 06:59:26.945596 21113 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:26.947772 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 21113
I20260927 06:59:26.947986 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:26.948454 388 external_mini_cluster.cc:1428] Setting key b747c6f3382ff1e3b94e0c37a169e634
I20260927 06:59:26.949386 21180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 06:59:26 dist-test-slave-f892 krb5kdc[21100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 06:59:26.963272 21180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4: Bootstrap starting.
I20260927 06:59:26.966324 21180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:26.967356 21180 log.cc:824] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:26.970201 21180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4: No bootstrap required, opened a new log
I20260927 06:59:26.970786 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.950162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39961 (local address 127.0.0.1:43781)
0927 06:59:26.950880 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:26.950909 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:26.950949 (+ 40us) server_negotiation.cc:407] Connection header received
0927 06:59:26.952462 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:26.952480 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:26.952860 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:26.953173 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:26.954926 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.956274 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.958109 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.958723 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.958849 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.963493 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:26.963502 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:26.963515 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:26.963549 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:26.966694 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:26.967431 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:26.967437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:26.967443 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:26.967643 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:26.968054 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:26.968059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:26.968062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:26.968583 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:26.968892 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:26.969348 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 06:59:26.969744 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":341,"threads_started":1}
I20260927 06:59:26.974453 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:26.979269 21180 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf24fd8fa9d424fa2d4eea6c243d0d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } }
I20260927 06:59:26.979701 21180 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:26.979837 21180 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf24fd8fa9d424fa2d4eea6c243d0d4, State: Initialized, Role: FOLLOWER
I20260927 06:59:26.980443 21180 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf24fd8fa9d424fa2d4eea6c243d0d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } }
I20260927 06:59:26.980686 21180 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:26.980845 21180 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:26.981045 21180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:26.983387 21180 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf24fd8fa9d424fa2d4eea6c243d0d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } }
I20260927 06:59:26.983942 21180 leader_election.cc:302] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbf24fd8fa9d424fa2d4eea6c243d0d4; no voters:
I20260927 06:59:26.984615 21180 leader_election.cc:288] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:26.984723 21190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:26.986801 21180 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:26.989677 21190 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4 [term 1 LEADER]: Becoming Leader. State: Replica: bbf24fd8fa9d424fa2d4eea6c243d0d4, State: Running, Role: LEADER
I20260927 06:59:26.990391 21190 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf24fd8fa9d424fa2d4eea6c243d0d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } }
I20260927 06:59:26.993304 21191 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbf24fd8fa9d424fa2d4eea6c243d0d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf24fd8fa9d424fa2d4eea6c243d0d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } } }
I20260927 06:59:26.993707 21191 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:26.997644 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.979237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44323 (local address 127.0.0.1:43781)
0927 06:59:26.979425 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:26.979430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:26.979834 (+ 404us) server_negotiation.cc:407] Connection header received
0927 06:59:26.980020 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:26.980026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:26.980098 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:26.980204 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:26.981575 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.982421 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.983670 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.984348 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.984511 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.988358 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:26.988369 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:26.988380 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:26.988432 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:26.992684 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:26.993677 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:26.993686 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:26.993690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:26.993934 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:26.994520 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:26.994539 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:26.994545 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:26.995052 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:26.995393 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:26.996847 (+ 1454us) server_negotiation.cc:299] Negotiation successful
0927 06:59:26.997134 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 06:59:27.001763 21192 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbf24fd8fa9d424fa2d4eea6c243d0d4. Latest consensus state: current_term: 1 leader_uuid: "bbf24fd8fa9d424fa2d4eea6c243d0d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf24fd8fa9d424fa2d4eea6c243d0d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } } }
I20260927 06:59:27.002069 21192 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:27.002125 21198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:27.007769 21198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:27.016675 21198 catalog_manager.cc:1347] Generated new cluster ID: ae93d9312bfc480d987a1cb2f9cdef58
I20260927 06:59:27.016798 21198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:27.052536 21198 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:27.053906 21198 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:27.063555 21198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbf24fd8fa9d424fa2d4eea6c243d0d4: Generated new TSK 0
I20260927 06:59:27.064303 21198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:27.069938 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 21113
I20260927 06:59:27.070947 21175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:27.162410 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 21113
2026-09-27T06:59:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:27 dist-test-slave-f892 krb5kdc[21213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:27 dist-test-slave-f892 krb5kdc[21213](info): set up 2 sockets
Sep 27 06:59:27 dist-test-slave-f892 krb5kdc[21213](info): commencing operation
krb5kdc: starting...
W20260927 06:59:28.254417 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:28 dist-test-slave-f892 krb5kdc[21213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 06:59:28.492475 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:28.807073 21226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:28.807423 21226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:28.807602 21226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:28.814770 21226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:28.814930 21226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:28.815047 21226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:28.815155 21226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:28.815243 21226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:28.824944 21226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:28.826231 21226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:28.827494 21226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:28.838766 21234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:28.838840 21231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:28.838862 21232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:28.839546 21226 server_base.cc:1034] running on GCE node
I20260927 06:59:28.840305 21226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:28.841625 21226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:28.842842 21226 hybrid_clock.cc:648] HybridClock initialized: now 1790492368842803 us; error 50 us; skew 500 ppm
I20260927 06:59:28.843181 21226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:28 dist-test-slave-f892 krb5kdc[21213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:28.846243 21226 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 06:59:28.847594 21226 webserver.cc:460] Webserver started at http://127.0.0.1:39185/ using document root <none> and password file <none>
I20260927 06:59:28.848224 21226 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:28.848366 21226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:28.848658 21226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:28.851531 21226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "3da0817c9a604a18bde7e7dd4ab52b88"
format_stamp: "Formatted at 2026-09-27 06:59:28 on dist-test-slave-f892"
server_key: "9a63fe2260534689438e03de9d9c1e53"
server_key_iv: "64f961473fceb674761b1a9317f08144"
server_key_version: "encryptionkey@0"
I20260927 06:59:28.852257 21226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "3da0817c9a604a18bde7e7dd4ab52b88"
format_stamp: "Formatted at 2026-09-27 06:59:28 on dist-test-slave-f892"
server_key: "9a63fe2260534689438e03de9d9c1e53"
server_key_iv: "64f961473fceb674761b1a9317f08144"
server_key_version: "encryptionkey@0"
I20260927 06:59:28.857126 21226 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 06:59:28.860394 21241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:28.861562 21226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 06:59:28.861734 21226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "3da0817c9a604a18bde7e7dd4ab52b88"
format_stamp: "Formatted at 2026-09-27 06:59:28 on dist-test-slave-f892"
server_key: "9a63fe2260534689438e03de9d9c1e53"
server_key_iv: "64f961473fceb674761b1a9317f08144"
server_key_version: "encryptionkey@0"
I20260927 06:59:28.861919 21226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:28.892021 21226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:28.896399 21226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:28.896687 21226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:28.914629 21226 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38295
I20260927 06:59:28.914677 21292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38295 every 8 connection(s)
I20260927 06:59:28.915942 21226 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:28.919458 21293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:28.924417 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 21226
I20260927 06:59:28.924592 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:28.925087 388 external_mini_cluster.cc:1428] Setting key b049d4084a796ca369a429f4b7b63479
I20260927 06:59:28.930454 21293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88: Bootstrap starting.
I20260927 06:59:28.933274 21293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:28.934330 21293 log.cc:824] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:28.937207 21293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88: No bootstrap required, opened a new log
I20260927 06:59:28.937582 21296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:28.926870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51327 (local address 127.0.0.1:38295)
0927 06:59:28.927628 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:28.927638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:28.927675 (+ 37us) server_negotiation.cc:407] Connection header received
0927 06:59:28.929379 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:28.929398 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:28.929750 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:28.930143 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:28.930642 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:28.931890 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:28.933211 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:28.933928 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:28.934154 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:28.935325 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:28.935341 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:28.935359 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:28.935929 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:28.936010 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:28.936033 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:28.936116 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:28.936366 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 06:59:28.936622 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":339,"threads_started":1}
I20260927 06:59:28.940973 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:28.946748 21293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da0817c9a604a18bde7e7dd4ab52b88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260927 06:59:28.947228 21293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:28.947347 21293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da0817c9a604a18bde7e7dd4ab52b88, State: Initialized, Role: FOLLOWER
I20260927 06:59:28.947932 21293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da0817c9a604a18bde7e7dd4ab52b88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260927 06:59:28.948173 21293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:28.948309 21293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:28.948505 21293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:28.951572 21293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da0817c9a604a18bde7e7dd4ab52b88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260927 06:59:28.952122 21293 leader_election.cc:302] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3da0817c9a604a18bde7e7dd4ab52b88; no voters:
I20260927 06:59:28.952693 21296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:28.946107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54003 (local address 127.0.0.1:38295)
0927 06:59:28.946291 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:28.946297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:28.946599 (+ 302us) server_negotiation.cc:407] Connection header received
0927 06:59:28.946764 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:28.946768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:28.946849 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:28.946922 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:28.947570 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:28.948359 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:28.950290 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:28.950740 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:28.950831 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:28.951919 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:28.951923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:28.951926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:28.951947 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:28.951983 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:28.951997 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:28.952066 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:28.952237 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 06:59:28.952338 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 06:59:28.952904 21293 leader_election.cc:288] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:28.953001 21303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:28.953708 21303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88 [term 1 LEADER]: Becoming Leader. State: Replica: 3da0817c9a604a18bde7e7dd4ab52b88, State: Running, Role: LEADER
I20260927 06:59:28.954445 21303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da0817c9a604a18bde7e7dd4ab52b88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260927 06:59:28.956517 21303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3da0817c9a604a18bde7e7dd4ab52b88. Latest consensus state: current_term: 1 leader_uuid: "3da0817c9a604a18bde7e7dd4ab52b88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da0817c9a604a18bde7e7dd4ab52b88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } } }
I20260927 06:59:28.956939 21303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:28.957317 21305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3da0817c9a604a18bde7e7dd4ab52b88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da0817c9a604a18bde7e7dd4ab52b88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } } }
I20260927 06:59:28.957648 21305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:28.960983 21293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:28.961150 21307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:28.967371 21307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:28.977342 21307 catalog_manager.cc:1347] Generated new cluster ID: 826bad5e370048fc9906f80d627a3028
I20260927 06:59:28.977488 21307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:29.005580 21307 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:29.007021 21307 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:29.015524 21307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3da0817c9a604a18bde7e7dd4ab52b88: Generated new TSK 0
I20260927 06:59:29.016287 21307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:29.024673 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 21226
W20260927 06:59:29.025851 21288 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51327
W20260927 06:59:29.026536 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 21226: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:59:29.026686 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 21226
2026-09-27T06:59:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1863 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:29 dist-test-slave-f892 krb5kdc[21324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:29 dist-test-slave-f892 krb5kdc[21324](info): set up 2 sockets
Sep 27 06:59:29 dist-test-slave-f892 krb5kdc[21324](info): commencing operation
krb5kdc: starting...
W20260927 06:59:30.126660 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:30 dist-test-slave-f892 krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 06:59:30.364993 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:30.680929 21337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:30.681205 21337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:30.681334 21337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:30.688223 21337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:30.688324 21337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:30.688367 21337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:30.688446 21337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:30.688529 21337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:30.698167 21337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:30.699427 21337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:30.700672 21337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:30.712366 21343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:30.712414 21342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:30.712330 21345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:30.713218 21337 server_base.cc:1034] running on GCE node
I20260927 06:59:30.713896 21337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:30.715324 21337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:30.716518 21337 hybrid_clock.cc:648] HybridClock initialized: now 1790492370716460 us; error 52 us; skew 500 ppm
I20260927 06:59:30.716862 21337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:30 dist-test-slave-f892 krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:30.723410 21337 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 06:59:30.724815 21337 webserver.cc:460] Webserver started at http://127.0.0.1:36169/ using document root <none> and password file <none>
I20260927 06:59:30.725485 21337 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:30.725632 21337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:30.725936 21337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:30.728880 21337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "01c21b0e11f7468583236ef174cdb4e7"
format_stamp: "Formatted at 2026-09-27 06:59:30 on dist-test-slave-f892"
server_key: "434d452d57863639abecdf5b61ce3b36"
server_key_iv: "480d43b082d2d227f5bd9109501c0967"
server_key_version: "encryptionkey@0"
I20260927 06:59:30.729616 21337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "01c21b0e11f7468583236ef174cdb4e7"
format_stamp: "Formatted at 2026-09-27 06:59:30 on dist-test-slave-f892"
server_key: "434d452d57863639abecdf5b61ce3b36"
server_key_iv: "480d43b082d2d227f5bd9109501c0967"
server_key_version: "encryptionkey@0"
I20260927 06:59:30.734412 21337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260927 06:59:30.737521 21352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:30.738776 21337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:59:30.738952 21337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "01c21b0e11f7468583236ef174cdb4e7"
format_stamp: "Formatted at 2026-09-27 06:59:30 on dist-test-slave-f892"
server_key: "434d452d57863639abecdf5b61ce3b36"
server_key_iv: "480d43b082d2d227f5bd9109501c0967"
server_key_version: "encryptionkey@0"
I20260927 06:59:30.739135 21337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:30.744529 21337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:30.748222 21337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:30.748514 21337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:30.766557 21337 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38937
I20260927 06:59:30.766630 21403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38937 every 8 connection(s)
I20260927 06:59:30.767829 21337 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:30.771204 21404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:30.777297 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 21337
I20260927 06:59:30.777568 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:30.777987 388 external_mini_cluster.cc:1428] Setting key 69676f077dac1c1381c6f5714be4111c
I20260927 06:59:30.781370 21404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7: Bootstrap starting.
I20260927 06:59:30.784514 21404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:30.785229 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:30.779673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53221 (local address 127.0.0.1:38937)
0927 06:59:30.780413 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:30.780424 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:30.780477 (+ 53us) server_negotiation.cc:407] Connection header received
0927 06:59:30.782212 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:30.782233 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:30.782520 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:30.782872 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:30.783801 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:30.783807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:30.783818 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:30.783831 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:30.783878 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:30.783895 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:30.784082 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:30.784209 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 06:59:30.784370 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":327,"threads_started":1}
I20260927 06:59:30.785599 21404 log.cc:824] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:30.788368 21404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7: No bootstrap required, opened a new log
I20260927 06:59:30.788585 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:30.796397 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:30.793706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36627 (local address 127.0.0.1:38937)
0927 06:59:30.793933 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:30.793953 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:30.794354 (+ 401us) server_negotiation.cc:407] Connection header received
0927 06:59:30.794572 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:30.794575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:30.794623 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:30.794707 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:30.795728 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:30.795732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:30.795735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:30.795749 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:30.795788 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:30.795805 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:30.795898 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:30.796036 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 06:59:30.796082 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 06:59:30.799383 21404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c21b0e11f7468583236ef174cdb4e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38937 } }
I20260927 06:59:30.799918 21404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:30.800042 21404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c21b0e11f7468583236ef174cdb4e7, State: Initialized, Role: FOLLOWER
I20260927 06:59:30.800642 21404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c21b0e11f7468583236ef174cdb4e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38937 } }
I20260927 06:59:30.800864 21404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:30.800973 21404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:30.801098 21404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:30.803442 21404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c21b0e11f7468583236ef174cdb4e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38937 } }
I20260927 06:59:30.803882 21404 leader_election.cc:302] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01c21b0e11f7468583236ef174cdb4e7; no voters:
I20260927 06:59:30.804390 21404 leader_election.cc:288] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:30.804487 21414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:30.805248 21414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 01c21b0e11f7468583236ef174cdb4e7, State: Running, Role: LEADER
I20260927 06:59:30.805826 21414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c21b0e11f7468583236ef174cdb4e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38937 } }
I20260927 06:59:30.806804 21404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:30.808732 21415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01c21b0e11f7468583236ef174cdb4e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c21b0e11f7468583236ef174cdb4e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38937 } } }
I20260927 06:59:30.809031 21416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01c21b0e11f7468583236ef174cdb4e7. Latest consensus state: current_term: 1 leader_uuid: "01c21b0e11f7468583236ef174cdb4e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c21b0e11f7468583236ef174cdb4e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38937 } } }
I20260927 06:59:30.809226 21415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:30.810449 21416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:30.811506 21421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:30.816313 21421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:30.824564 21421 catalog_manager.cc:1347] Generated new cluster ID: e0ea6c0fbeb44132827160ee9744e3e5
I20260927 06:59:30.824659 21421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:30.835632 21421 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:30.836004 21421 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:30.843989 21421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01c21b0e11f7468583236ef174cdb4e7: Generated new TSK 0
I20260927 06:59:30.844655 21421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:30.853695 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 21337
W20260927 06:59:30.854707 21399 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53221
W20260927 06:59:30.855397 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 21337: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:59:30.855504 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 21337
2026-09-27T06:59:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1828 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:30 dist-test-slave-f892 krb5kdc[21435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:30 dist-test-slave-f892 krb5kdc[21435](info): set up 2 sockets
Sep 27 06:59:30 dist-test-slave-f892 krb5kdc[21435](info): commencing operation
krb5kdc: starting...
W20260927 06:59:31.946400 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:32 dist-test-slave-f892 krb5kdc[21435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 06:59:32.181191 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:32.496467 21448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:32.496709 21448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:32.496893 21448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:32.503754 21448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:32.503844 21448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:32.503942 21448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:32.504046 21448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:32.504122 21448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:32.514411 21448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:32.515684 21448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:32.516947 21448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:32.528227 21453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:32.528285 21454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:32.528313 21456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:32.528970 21448 server_base.cc:1034] running on GCE node
I20260927 06:59:32.529663 21448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:32.530941 21448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:32.532155 21448 hybrid_clock.cc:648] HybridClock initialized: now 1790492372532110 us; error 35 us; skew 500 ppm
I20260927 06:59:32.532472 21448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:32 dist-test-slave-f892 krb5kdc[21435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:32.535395 21448 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 06:59:32.536674 21448 webserver.cc:460] Webserver started at http://127.0.0.1:46447/ using document root <none> and password file <none>
I20260927 06:59:32.537282 21448 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:32.537421 21448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:32.537699 21448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:32.540392 21448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "556bf5b9f2e44aac98d405cda855a59c"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-f892"
server_key: "34115fc2df87bdd378689a82bd3e7a46"
server_key_iv: "d49b07cf6ac1c605c246a96d55236cd6"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.541095 21448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "556bf5b9f2e44aac98d405cda855a59c"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-f892"
server_key: "34115fc2df87bdd378689a82bd3e7a46"
server_key_iv: "d49b07cf6ac1c605c246a96d55236cd6"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.545842 21448 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260927 06:59:32.548954 21463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.550128 21448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 06:59:32.550299 21448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "556bf5b9f2e44aac98d405cda855a59c"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-f892"
server_key: "34115fc2df87bdd378689a82bd3e7a46"
server_key_iv: "d49b07cf6ac1c605c246a96d55236cd6"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.550475 21448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:32.563033 21448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:32.567011 21448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:32.567286 21448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:32.584689 21448 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42177
I20260927 06:59:32.584725 21514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42177 every 8 connection(s)
I20260927 06:59:32.585893 21448 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:32.589270 21515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:32.591078 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 21448
I20260927 06:59:32.591264 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:32.591715 388 external_mini_cluster.cc:1428] Setting key 1e3b75e8f5ad97f95242b0a89714506c
I20260927 06:59:32.603124 21515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c: Bootstrap starting.
Sep 27 06:59:32 dist-test-slave-f892 krb5kdc[21435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 06:59:32.605911 21515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:32.607224 21515 log.cc:824] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:32.610332 21515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c: No bootstrap required, opened a new log
I20260927 06:59:32.614507 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.593378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:41233 (local address 127.0.0.1:42177)
0927 06:59:32.594146 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.594159 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.594210 (+ 51us) server_negotiation.cc:407] Connection header received
0927 06:59:32.595882 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.595909 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.596427 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.596840 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.598091 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.600053 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.601258 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.602271 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.602453 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.606259 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.606271 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.606295 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.606355 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.610039 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.610840 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.610848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.610861 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.611077 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.611493 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.611498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.611501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.612159 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.612494 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.612856 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.613186 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":307,"threads_started":1}
I20260927 06:59:32.618227 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:32.619593 21515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556bf5b9f2e44aac98d405cda855a59c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42177 } }
I20260927 06:59:32.619926 21515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:32.620018 21515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556bf5b9f2e44aac98d405cda855a59c, State: Initialized, Role: FOLLOWER
I20260927 06:59:32.620591 21515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556bf5b9f2e44aac98d405cda855a59c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42177 } }
I20260927 06:59:32.620872 21515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:32.621011 21515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:32.621188 21515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:32.624109 21515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556bf5b9f2e44aac98d405cda855a59c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42177 } }
I20260927 06:59:32.624648 21515 leader_election.cc:302] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 556bf5b9f2e44aac98d405cda855a59c; no voters:
I20260927 06:59:32.625418 21515 leader_election.cc:288] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:32.625540 21525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:32.627807 21515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:32.632494 21525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c [term 1 LEADER]: Becoming Leader. State: Replica: 556bf5b9f2e44aac98d405cda855a59c, State: Running, Role: LEADER
I20260927 06:59:32.633229 21525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556bf5b9f2e44aac98d405cda855a59c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42177 } }
I20260927 06:59:32.636337 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.622393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:47985 (local address 127.0.0.1:42177)
0927 06:59:32.622610 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.622619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.623019 (+ 400us) server_negotiation.cc:407] Connection header received
0927 06:59:32.623167 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.623171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.623223 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.623296 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.624944 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.626124 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.627526 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.628003 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.628150 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.630680 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.630684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.630687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.630713 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.633502 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.634303 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.634309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.634312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.634448 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.634850 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.634855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.634858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.635178 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.635459 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.635825 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.635993 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 06:59:32.639300 21527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 556bf5b9f2e44aac98d405cda855a59c. Latest consensus state: current_term: 1 leader_uuid: "556bf5b9f2e44aac98d405cda855a59c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556bf5b9f2e44aac98d405cda855a59c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42177 } } }
I20260927 06:59:32.639647 21527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:32.645246 21526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "556bf5b9f2e44aac98d405cda855a59c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556bf5b9f2e44aac98d405cda855a59c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42177 } } }
I20260927 06:59:32.645731 21526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:32.645852 21534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:32.650669 21534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:32.658434 21534 catalog_manager.cc:1347] Generated new cluster ID: 001dfd6e37a8464e9cedd14cd39f2bb9
I20260927 06:59:32.658540 21534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:32.686480 21534 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:32.688273 21534 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:32.701009 21534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 556bf5b9f2e44aac98d405cda855a59c: Generated new TSK 0
I20260927 06:59:32.701980 21534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:32.710708 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 21448
I20260927 06:59:32.712122 21510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:32.802884 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 21448
W20260927 06:59:32.819535 21433 connection.cc:535] client connection to 127.0.0.1:42177 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42177): Connection reset by peer (error 104)
2026-09-27T06:59:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1948 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:32 dist-test-slave-f892 krb5kdc[21547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:32 dist-test-slave-f892 krb5kdc[21547](info): set up 2 sockets
Sep 27 06:59:32 dist-test-slave-f892 krb5kdc[21547](info): commencing operation
krb5kdc: starting...
W20260927 06:59:33.894362 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:34 dist-test-slave-f892 krb5kdc[21547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 06:59:34.128791 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44635
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:34.448043 21560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:34.448300 21560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:34.448387 21560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:34.455291 21560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:34.455384 21560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:34.455426 21560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:34.455487 21560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:34.455526 21560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:34.464507 21560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:34.465590 21560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:34.466789 21560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:34.477747 21565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:34.477792 21566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:34.477696 21568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:34.478574 21560 server_base.cc:1034] running on GCE node
I20260927 06:59:34.479275 21560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:34.480537 21560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:34.481767 21560 hybrid_clock.cc:648] HybridClock initialized: now 1790492374481727 us; error 57 us; skew 500 ppm
I20260927 06:59:34.482151 21560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:34 dist-test-slave-f892 krb5kdc[21547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:34.484982 21560 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 06:59:34.486268 21560 webserver.cc:460] Webserver started at http://127.0.0.1:40651/ using document root <none> and password file <none>
I20260927 06:59:34.486874 21560 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:34.487033 21560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:34.487337 21560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:34.490077 21560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "ae91f3fe4db347fa9ab95cd22bda1b92"
format_stamp: "Formatted at 2026-09-27 06:59:34 on dist-test-slave-f892"
server_key: "b3b8ce5cb506a5f9d3a24ed225dbb1ef"
server_key_iv: "4e83907588feb4ff00e5ec04cc702d66"
server_key_version: "encryptionkey@0"
I20260927 06:59:34.490845 21560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "ae91f3fe4db347fa9ab95cd22bda1b92"
format_stamp: "Formatted at 2026-09-27 06:59:34 on dist-test-slave-f892"
server_key: "b3b8ce5cb506a5f9d3a24ed225dbb1ef"
server_key_iv: "4e83907588feb4ff00e5ec04cc702d66"
server_key_version: "encryptionkey@0"
I20260927 06:59:34.495518 21560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 06:59:34.498591 21575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:34.499667 21560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 06:59:34.499836 21560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "ae91f3fe4db347fa9ab95cd22bda1b92"
format_stamp: "Formatted at 2026-09-27 06:59:34 on dist-test-slave-f892"
server_key: "b3b8ce5cb506a5f9d3a24ed225dbb1ef"
server_key_iv: "4e83907588feb4ff00e5ec04cc702d66"
server_key_version: "encryptionkey@0"
I20260927 06:59:34.500008 21560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:34.525341 21560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:34.529769 21560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:34.530079 21560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:34.547652 21560 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44635
I20260927 06:59:34.547696 21626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44635 every 8 connection(s)
I20260927 06:59:34.548974 21560 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:34.551362 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 21560
I20260927 06:59:34.551548 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:34.552073 388 external_mini_cluster.cc:1428] Setting key 9992e4769f2c8fd3f98864f80ff19bc5
I20260927 06:59:34.553004 21627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:34.564728 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:34.553768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:32895 (local address 127.0.0.1:44635)
0927 06:59:34.554511 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:34.554522 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:34.554560 (+ 38us) server_negotiation.cc:407] Connection header received
0927 06:59:34.556025 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:34.556045 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:34.556405 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:34.556724 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:34.557290 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:34.558677 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:34.560524 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:34.561134 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:34.561261 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:34.562577 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:34.562593 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:34.562606 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:34.563119 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:34.563200 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:34.563224 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:34.563337 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:34.563656 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 06:59:34.563834 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":357,"threads_started":1}
I20260927 06:59:34.566246 21627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92: Bootstrap starting.
I20260927 06:59:34.568514 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:34.569360 21627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:34.570324 21627 log.cc:824] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:34.572914 21627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92: No bootstrap required, opened a new log
I20260927 06:59:34.579978 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:34.574084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:37177 (local address 127.0.0.1:44635)
0927 06:59:34.574212 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:34.574215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:34.574688 (+ 473us) server_negotiation.cc:407] Connection header received
0927 06:59:34.574894 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:34.574897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:34.574949 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:34.575045 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:34.575513 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:34.576261 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:34.577536 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:34.578044 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:34.578200 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:34.579286 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:34.579291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:34.579294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:34.579312 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:34.579345 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:34.579360 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:34.579483 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:34.579672 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 06:59:34.579719 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 06:59:34.580431 21627 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae91f3fe4db347fa9ab95cd22bda1b92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44635 } }
I20260927 06:59:34.580792 21627 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:34.580919 21627 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae91f3fe4db347fa9ab95cd22bda1b92, State: Initialized, Role: FOLLOWER
I20260927 06:59:34.581542 21627 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae91f3fe4db347fa9ab95cd22bda1b92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44635 } }
I20260927 06:59:34.581738 21627 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:34.581835 21627 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:34.581944 21627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:34.584177 21627 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae91f3fe4db347fa9ab95cd22bda1b92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44635 } }
I20260927 06:59:34.584595 21627 leader_election.cc:302] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae91f3fe4db347fa9ab95cd22bda1b92; no voters:
I20260927 06:59:34.585028 21627 leader_election.cc:288] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:34.585170 21637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:34.585837 21637 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92 [term 1 LEADER]: Becoming Leader. State: Replica: ae91f3fe4db347fa9ab95cd22bda1b92, State: Running, Role: LEADER
I20260927 06:59:34.586300 21637 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae91f3fe4db347fa9ab95cd22bda1b92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44635 } }
I20260927 06:59:34.586988 21627 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:34.589008 21639 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae91f3fe4db347fa9ab95cd22bda1b92. Latest consensus state: current_term: 1 leader_uuid: "ae91f3fe4db347fa9ab95cd22bda1b92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae91f3fe4db347fa9ab95cd22bda1b92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44635 } } }
I20260927 06:59:34.589439 21639 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:34.593945 21638 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae91f3fe4db347fa9ab95cd22bda1b92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae91f3fe4db347fa9ab95cd22bda1b92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44635 } } }
I20260927 06:59:34.594337 21638 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:34.594728 21644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:34.599717 21644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:34.608364 21644 catalog_manager.cc:1347] Generated new cluster ID: caa445176b9343c3bafc91e664594ec1
I20260927 06:59:34.608485 21644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:34.616943 21644 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:34.618161 21644 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:34.625005 21644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae91f3fe4db347fa9ab95cd22bda1b92: Generated new TSK 0
I20260927 06:59:34.625953 21644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:34.636454 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 21560
W20260927 06:59:34.637974 21622 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.18:32895
W20260927 06:59:34.638734 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 21560: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:59:34.638878 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 21560
W20260927 06:59:34.655390 21545 connection.cc:535] client connection to 127.0.0.1:44635 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44635): Connection reset by peer (error 104)
2026-09-27T06:59:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1835 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:34 dist-test-slave-f892 krb5kdc[21658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:34 dist-test-slave-f892 krb5kdc[21658](info): set up 2 sockets
Sep 27 06:59:34 dist-test-slave-f892 krb5kdc[21658](info): commencing operation
krb5kdc: starting...
W20260927 06:59:35.730367 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:35 dist-test-slave-f892 krb5kdc[21658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 06:59:35.971138 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:36.291558 21671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:36.291843 21671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:36.291940 21671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:36.299257 21671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:36.299374 21671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:36.299428 21671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:36.299485 21671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:36.299528 21671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:36.309147 21671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:36.310273 21671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:36.311447 21671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:36.322877 21676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:36.322979 21677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:36.322975 21679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:36.323637 21671 server_base.cc:1034] running on GCE node
I20260927 06:59:36.324353 21671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:36.325567 21671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:36.326787 21671 hybrid_clock.cc:648] HybridClock initialized: now 1790492376326734 us; error 43 us; skew 500 ppm
I20260927 06:59:36.327114 21671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:36 dist-test-slave-f892 krb5kdc[21658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:36.333273 21671 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 06:59:36.334654 21671 webserver.cc:460] Webserver started at http://127.0.0.1:36047/ using document root <none> and password file <none>
I20260927 06:59:36.335278 21671 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:36.335419 21671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:36.335713 21671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:36.338522 21671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "3deb3edef8844f7ea7affb6b03179ae9"
format_stamp: "Formatted at 2026-09-27 06:59:36 on dist-test-slave-f892"
server_key: "d5873b02d6b6535c1178d4f73569da96"
server_key_iv: "8607fcc08fd6195343feed56ceef594f"
server_key_version: "encryptionkey@0"
I20260927 06:59:36.339231 21671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "3deb3edef8844f7ea7affb6b03179ae9"
format_stamp: "Formatted at 2026-09-27 06:59:36 on dist-test-slave-f892"
server_key: "d5873b02d6b6535c1178d4f73569da96"
server_key_iv: "8607fcc08fd6195343feed56ceef594f"
server_key_version: "encryptionkey@0"
I20260927 06:59:36.344628 21671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 06:59:36.348079 21686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:36.349253 21671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 06:59:36.349432 21671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "3deb3edef8844f7ea7affb6b03179ae9"
format_stamp: "Formatted at 2026-09-27 06:59:36 on dist-test-slave-f892"
server_key: "d5873b02d6b6535c1178d4f73569da96"
server_key_iv: "8607fcc08fd6195343feed56ceef594f"
server_key_version: "encryptionkey@0"
I20260927 06:59:36.349639 21671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:36.355026 21671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:36.358559 21671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:36.358834 21671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:36.376652 21671 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44583
I20260927 06:59:36.376700 21737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44583 every 8 connection(s)
I20260927 06:59:36.377955 21671 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:36.381559 21738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:36.383756 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 21671
I20260927 06:59:36.383965 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:36.384393 388 external_mini_cluster.cc:1428] Setting key ffad1128fc9c79763b52fedd1f43f0bc
I20260927 06:59:36.392230 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:36.386143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:48029 (local address 127.0.0.1:44583)
0927 06:59:36.387016 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:36.387028 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:36.387070 (+ 42us) server_negotiation.cc:407] Connection header received
0927 06:59:36.389006 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:36.389030 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:36.389348 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:36.389771 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:36.390752 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:36.390757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:36.390769 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:36.390783 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:36.390836 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:36.390859 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:36.391079 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:36.391215 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 06:59:36.391395 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":333,"threads_started":1}
I20260927 06:59:36.395843 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:36.396051 21738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9: Bootstrap starting.
I20260927 06:59:36.399151 21738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:36.400462 21738 log.cc:824] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:36.403654 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:36.400246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:42475 (local address 127.0.0.1:44583)
0927 06:59:36.400399 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:36.400403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:36.401746 (+ 1343us) server_negotiation.cc:407] Connection header received
0927 06:59:36.401911 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:36.401915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:36.401957 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:36.402105 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:36.402999 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:36.403004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:36.403007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:36.403021 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:36.403057 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:36.403070 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:36.403155 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:36.403299 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 06:59:36.403346 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 06:59:36.403805 21738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9: No bootstrap required, opened a new log
I20260927 06:59:36.411429 21738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3deb3edef8844f7ea7affb6b03179ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44583 } }
I20260927 06:59:36.411778 21738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:36.411898 21738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3deb3edef8844f7ea7affb6b03179ae9, State: Initialized, Role: FOLLOWER
I20260927 06:59:36.412640 21738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3deb3edef8844f7ea7affb6b03179ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44583 } }
I20260927 06:59:36.412922 21738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:36.413076 21738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:36.413291 21738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:36.415474 21738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3deb3edef8844f7ea7affb6b03179ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44583 } }
I20260927 06:59:36.415905 21738 leader_election.cc:302] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3deb3edef8844f7ea7affb6b03179ae9; no voters:
I20260927 06:59:36.416425 21738 leader_election.cc:288] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:36.416543 21748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:36.417312 21748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9 [term 1 LEADER]: Becoming Leader. State: Replica: 3deb3edef8844f7ea7affb6b03179ae9, State: Running, Role: LEADER
I20260927 06:59:36.417891 21748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3deb3edef8844f7ea7affb6b03179ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44583 } }
I20260927 06:59:36.418689 21738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:36.421494 21749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3deb3edef8844f7ea7affb6b03179ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3deb3edef8844f7ea7affb6b03179ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44583 } } }
I20260927 06:59:36.421808 21749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:36.422201 21750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3deb3edef8844f7ea7affb6b03179ae9. Latest consensus state: current_term: 1 leader_uuid: "3deb3edef8844f7ea7affb6b03179ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3deb3edef8844f7ea7affb6b03179ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44583 } } }
I20260927 06:59:36.422549 21750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:36.427062 21756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:36.432490 21756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:36.440976 21756 catalog_manager.cc:1347] Generated new cluster ID: 60000f785c7245379162fec339015bf7
I20260927 06:59:36.441092 21756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:36.451874 21756 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:36.452235 21756 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:36.462415 21756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3deb3edef8844f7ea7affb6b03179ae9: Generated new TSK 0
I20260927 06:59:36.463102 21756 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:36.471670 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 21671
W20260927 06:59:36.472858 21733 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.18:48029
W20260927 06:59:36.473534 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 21671: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:59:36.473673 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 21671
2026-09-27T06:59:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1834 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:36 dist-test-slave-f892 krb5kdc[21769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:36 dist-test-slave-f892 krb5kdc[21769](info): set up 2 sockets
Sep 27 06:59:36 dist-test-slave-f892 krb5kdc[21769](info): commencing operation
krb5kdc: starting...
W20260927 06:59:37.558322 388 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:37 dist-test-slave-f892 krb5kdc[21769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 06:59:37.795195 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44275
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:38.108533 21782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:38.108806 21782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:38.109014 21782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:38.116075 21782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:38.116194 21782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:38.116295 21782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:38.116394 21782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:38.116459 21782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:38.126055 21782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44275
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:38.127292 21782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:38.128535 21782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:38.140201 21788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:38.140196 21790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:38.140723 21787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:38.141049 21782 server_base.cc:1034] running on GCE node
I20260927 06:59:38.141791 21782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:38.143167 21782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:38.144364 21782 hybrid_clock.cc:648] HybridClock initialized: now 1790492378144317 us; error 37 us; skew 500 ppm
I20260927 06:59:38.144690 21782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:38 dist-test-slave-f892 krb5kdc[21769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:38.148134 21782 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 06:59:38.149457 21782 webserver.cc:460] Webserver started at http://127.0.0.1:39399/ using document root <none> and password file <none>
I20260927 06:59:38.150134 21782 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:38.150239 21782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:38.150502 21782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:38.153264 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "d55d6f9bbdb943d6a7dd546dfba88c8a"
format_stamp: "Formatted at 2026-09-27 06:59:38 on dist-test-slave-f892"
server_key: "9952b4a69e3f789a9a59ed4d38c92efc"
server_key_iv: "1d83dccf8d9103b0a2e656afc1d7592a"
server_key_version: "encryptionkey@0"
I20260927 06:59:38.153932 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "d55d6f9bbdb943d6a7dd546dfba88c8a"
format_stamp: "Formatted at 2026-09-27 06:59:38 on dist-test-slave-f892"
server_key: "9952b4a69e3f789a9a59ed4d38c92efc"
server_key_iv: "1d83dccf8d9103b0a2e656afc1d7592a"
server_key_version: "encryptionkey@0"
I20260927 06:59:38.158666 21782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 06:59:38.161862 21797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:38.163058 21782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:59:38.163233 21782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "d55d6f9bbdb943d6a7dd546dfba88c8a"
format_stamp: "Formatted at 2026-09-27 06:59:38 on dist-test-slave-f892"
server_key: "9952b4a69e3f789a9a59ed4d38c92efc"
server_key_iv: "1d83dccf8d9103b0a2e656afc1d7592a"
server_key_version: "encryptionkey@0"
I20260927 06:59:38.163427 21782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:38.174861 21782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:38.178934 21782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:38.179196 21782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:38.196024 21782 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44275
I20260927 06:59:38.196058 21848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44275 every 8 connection(s)
I20260927 06:59:38.197260 21782 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:38.200625 21849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:38.207185 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 21782
I20260927 06:59:38.207376 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:38.207775 388 external_mini_cluster.cc:1428] Setting key b3789e8cb41552b0b073c76712e304d6
I20260927 06:59:38.210825 21849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a: Bootstrap starting.
I20260927 06:59:38.214558 21849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:38.215772 21849 log.cc:824] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:38.218489 21849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a: No bootstrap required, opened a new log
Sep 27 06:59:38 dist-test-slave-f892 krb5kdc[21769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 06:59:38.227293 21849 raft_consensus.cc:348] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d55d6f9bbdb943d6a7dd546dfba88c8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44275 } }
I20260927 06:59:38.227702 21849 raft_consensus.cc:374] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:38.227824 21849 raft_consensus.cc:729] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d55d6f9bbdb943d6a7dd546dfba88c8a, State: Initialized, Role: FOLLOWER
I20260927 06:59:38.228457 21849 consensus_queue.cc:260] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d55d6f9bbdb943d6a7dd546dfba88c8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44275 } }
I20260927 06:59:38.228744 21849 raft_consensus.cc:388] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:38.228893 21849 raft_consensus.cc:482] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:38.229104 21849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:38.231391 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:38.209489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:36259 (local address 127.0.0.1:44275)
0927 06:59:38.210257 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:38.210273 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:38.210320 (+ 47us) server_negotiation.cc:407] Connection header received
0927 06:59:38.212750 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:38.212781 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:38.213250 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:38.213644 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:38.215000 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:38.216826 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:38.218077 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:38.219091 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:38.219266 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:38.223485 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:38.223492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:38.223508 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:38.223541 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:38.226600 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:38.227452 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:38.227457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:38.227462 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:38.227609 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:38.228224 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:38.228229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:38.228232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:38.228899 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:38.229237 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:38.229739 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 06:59:38.230081 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":288,"threads_started":1}
I20260927 06:59:38.232173 21849 raft_consensus.cc:504] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d55d6f9bbdb943d6a7dd546dfba88c8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44275 } }
I20260927 06:59:38.232633 21849 leader_election.cc:302] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d55d6f9bbdb943d6a7dd546dfba88c8a; no voters:
I20260927 06:59:38.233237 21849 leader_election.cc:288] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:38.233386 21854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:38.234845 21854 raft_consensus.cc:686] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a [term 1 LEADER]: Becoming Leader. State: Replica: d55d6f9bbdb943d6a7dd546dfba88c8a, State: Running, Role: LEADER
I20260927 06:59:38.235286 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:38.235430 21854 consensus_queue.cc:237] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d55d6f9bbdb943d6a7dd546dfba88c8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44275 } }
I20260927 06:59:38.235656 21849 sys_catalog.cc:564] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:38.238161 21856 sys_catalog.cc:455] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d55d6f9bbdb943d6a7dd546dfba88c8a. Latest consensus state: current_term: 1 leader_uuid: "d55d6f9bbdb943d6a7dd546dfba88c8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d55d6f9bbdb943d6a7dd546dfba88c8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44275 } } }
I20260927 06:59:38.238173 21855 sys_catalog.cc:455] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d55d6f9bbdb943d6a7dd546dfba88c8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d55d6f9bbdb943d6a7dd546dfba88c8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44275 } } }
I20260927 06:59:38.238603 21855 sys_catalog.cc:458] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:38.238603 21856 sys_catalog.cc:458] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:38.242537 21859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:38.251398 21859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:38.261392 21859 catalog_manager.cc:1347] Generated new cluster ID: c3db206a4632453e8af8193bb6ea0a26
I20260927 06:59:38.261497 21859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:38.267144 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:38.247417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:33815 (local address 127.0.0.1:44275)
0927 06:59:38.247592 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:38.247599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:38.250569 (+ 2970us) server_negotiation.cc:407] Connection header received
0927 06:59:38.250746 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:38.250752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:38.250830 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:38.250940 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:38.252918 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:38.254122 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:38.256003 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:38.256627 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:38.256728 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:38.260033 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:38.260039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:38.260043 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:38.260076 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:38.263766 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:38.264631 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:38.264636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:38.264640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:38.264759 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:38.265215 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:38.265220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:38.265224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:38.265573 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:38.265784 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:38.266544 (+ 760us) server_negotiation.cc:299] Negotiation successful
0927 06:59:38.266708 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 06:59:38.271618 21859 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:38.272799 21859 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:38.279579 21859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d55d6f9bbdb943d6a7dd546dfba88c8a: Generated new TSK 0
I20260927 06:59:38.280262 21859 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:38.291793 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 21782
I20260927 06:59:38.293730 21844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:38.381927 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 21782
W20260927 06:59:38.399422 21767 connection.cc:535] client connection to 127.0.0.1:44275 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44275): Connection reset by peer (error 104)
2026-09-27T06:59:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1908 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:38 dist-test-slave-f892 krb5kdc[21882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:38 dist-test-slave-f892 krb5kdc[21882](info): set up 2 sockets
Sep 27 06:59:38 dist-test-slave-f892 krb5kdc[21882](info): commencing operation
krb5kdc: starting...
W20260927 06:59:39.470404 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:39 dist-test-slave-f892 krb5kdc[21882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 06:59:39.700904 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:40.017621 21895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:40.017897 21895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:40.017988 21895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:40.025166 21895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:40.025269 21895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:40.025313 21895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:40.025368 21895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:40.025413 21895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:40.035076 21895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36351
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:40.036283 21895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:40.037470 21895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:40.048625 21900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:40.048588 21901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:40.048666 21903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:40.049515 21895 server_base.cc:1034] running on GCE node
I20260927 06:59:40.050288 21895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:40.051513 21895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:40.052726 21895 hybrid_clock.cc:648] HybridClock initialized: now 1790492380052689 us; error 45 us; skew 500 ppm
I20260927 06:59:40.053043 21895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:40 dist-test-slave-f892 krb5kdc[21882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:40.055943 21895 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 06:59:40.057245 21895 webserver.cc:460] Webserver started at http://127.0.0.1:42641/ using document root <none> and password file <none>
I20260927 06:59:40.057863 21895 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:40.057993 21895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:40.058306 21895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:40.061057 21895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "28772befd89140c3a0d2d0049da91690"
format_stamp: "Formatted at 2026-09-27 06:59:40 on dist-test-slave-f892"
server_key: "809823641a5533d9db412b425872566f"
server_key_iv: "cd1ae9407a250424f707351ed62da681"
server_key_version: "encryptionkey@0"
I20260927 06:59:40.061781 21895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "28772befd89140c3a0d2d0049da91690"
format_stamp: "Formatted at 2026-09-27 06:59:40 on dist-test-slave-f892"
server_key: "809823641a5533d9db412b425872566f"
server_key_iv: "cd1ae9407a250424f707351ed62da681"
server_key_version: "encryptionkey@0"
I20260927 06:59:40.066550 21895 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 06:59:40.069655 21910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:40.070852 21895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 06:59:40.071007 21895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "28772befd89140c3a0d2d0049da91690"
format_stamp: "Formatted at 2026-09-27 06:59:40 on dist-test-slave-f892"
server_key: "809823641a5533d9db412b425872566f"
server_key_iv: "cd1ae9407a250424f707351ed62da681"
server_key_version: "encryptionkey@0"
I20260927 06:59:40.071179 21895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:40.086160 21895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:40.089869 21895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:40.090229 21895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:40.107818 21895 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36351
I20260927 06:59:40.107856 21961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36351 every 8 connection(s)
I20260927 06:59:40.109167 21895 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:40.112775 21962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:40.113425 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 21895
I20260927 06:59:40.113592 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:40.114074 388 external_mini_cluster.cc:1428] Setting key aab2094e307f19f3f16b016872587c45
I20260927 06:59:40.124598 21962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690: Bootstrap starting.
I20260927 06:59:40.126902 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:40.115812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:49991 (local address 127.0.0.1:36351)
0927 06:59:40.116536 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:40.116548 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:40.116591 (+ 43us) server_negotiation.cc:407] Connection header received
0927 06:59:40.118869 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:40.118890 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:40.119271 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:40.119652 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:40.120148 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:40.121380 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:40.122742 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:40.123368 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:40.123555 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:40.124812 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:40.124819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:40.124832 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:40.125333 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:40.125399 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:40.125422 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:40.125571 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:40.125831 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 06:59:40.125999 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":307,"threads_started":1}
I20260927 06:59:40.127652 21962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:40.128782 21962 log.cc:824] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:40.130462 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:40.131522 21962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690: No bootstrap required, opened a new log
I20260927 06:59:40.140067 21962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28772befd89140c3a0d2d0049da91690" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36351 } }
I20260927 06:59:40.140506 21962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:40.140646 21962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28772befd89140c3a0d2d0049da91690, State: Initialized, Role: FOLLOWER
I20260927 06:59:40.141140 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:40.134735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:58475 (local address 127.0.0.1:36351)
0927 06:59:40.134888 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:40.134894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:40.135245 (+ 351us) server_negotiation.cc:407] Connection header received
0927 06:59:40.135378 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:40.135383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:40.135459 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:40.135566 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:40.136026 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:40.137178 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:40.138476 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:40.139168 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:40.139290 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:40.140313 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:40.140320 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:40.140324 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:40.140343 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:40.140386 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:40.140406 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:40.140533 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:40.140790 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 06:59:40.140840 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 06:59:40.141299 21962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28772befd89140c3a0d2d0049da91690" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36351 } }
I20260927 06:59:40.141543 21962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:40.141695 21962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:40.141877 21962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:40.143968 21962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28772befd89140c3a0d2d0049da91690" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36351 } }
I20260927 06:59:40.144425 21962 leader_election.cc:302] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28772befd89140c3a0d2d0049da91690; no voters:
I20260927 06:59:40.144946 21962 leader_election.cc:288] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:40.145069 21972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:40.145787 21972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690 [term 1 LEADER]: Becoming Leader. State: Replica: 28772befd89140c3a0d2d0049da91690, State: Running, Role: LEADER
I20260927 06:59:40.146365 21972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28772befd89140c3a0d2d0049da91690" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36351 } }
I20260927 06:59:40.146907 21962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:40.149663 21973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28772befd89140c3a0d2d0049da91690" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28772befd89140c3a0d2d0049da91690" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36351 } } }
I20260927 06:59:40.150341 21973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:40.150677 21974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28772befd89140c3a0d2d0049da91690. Latest consensus state: current_term: 1 leader_uuid: "28772befd89140c3a0d2d0049da91690" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28772befd89140c3a0d2d0049da91690" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36351 } } }
I20260927 06:59:40.151291 21974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:40.152472 21981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:40.157562 21981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:40.166150 21981 catalog_manager.cc:1347] Generated new cluster ID: 6152ad95a56944ea803449c2aa53f8c0
I20260927 06:59:40.166260 21981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:40.180480 21981 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:40.181711 21981 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:40.191610 21981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28772befd89140c3a0d2d0049da91690: Generated new TSK 0
I20260927 06:59:40.192449 21981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:40.205523 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 21895
W20260927 06:59:40.206662 21957 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.18:49991
W20260927 06:59:40.207396 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 21895: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:59:40.207543 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 21895
W20260927 06:59:40.224015 21880 connection.cc:535] client connection to 127.0.0.1:36351 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36351): Connection reset by peer (error 104)
2026-09-27T06:59:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1824 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:40 dist-test-slave-f892 krb5kdc[21993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:40 dist-test-slave-f892 krb5kdc[21993](info): set up 2 sockets
Sep 27 06:59:40 dist-test-slave-f892 krb5kdc[21993](info): commencing operation
krb5kdc: starting...
W20260927 06:59:41.297060 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:41 dist-test-slave-f892 krb5kdc[21993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 06:59:41.541534 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33187
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:41.854697 22006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:41.854948 22006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:41.855036 22006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:41.861971 22006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:41.862161 22006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:41.862231 22006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:41.862291 22006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:41.862334 22006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:41.871634 22006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33187
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:41.872736 22006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:41.873876 22006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:41.885236 22014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:41.885243 22012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:41.885537 22011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:41.886251 22006 server_base.cc:1034] running on GCE node
I20260927 06:59:41.886932 22006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:41.888167 22006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:41.889417 22006 hybrid_clock.cc:648] HybridClock initialized: now 1790492381889383 us; error 42 us; skew 500 ppm
I20260927 06:59:41.889767 22006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:41 dist-test-slave-f892 krb5kdc[21993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:41.897781 22006 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 06:59:41.899137 22006 webserver.cc:460] Webserver started at http://127.0.0.1:33967/ using document root <none> and password file <none>
I20260927 06:59:41.899786 22006 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:41.899948 22006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:41.900256 22006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:41.903149 22006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "93dc48f2bdde49f88fa7e9ca325a53a1"
format_stamp: "Formatted at 2026-09-27 06:59:41 on dist-test-slave-f892"
server_key: "360c27674c6015ea3625697053d335fd"
server_key_iv: "4ee0e97c756955a9a21a1a4562192b23"
server_key_version: "encryptionkey@0"
I20260927 06:59:41.903889 22006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "93dc48f2bdde49f88fa7e9ca325a53a1"
format_stamp: "Formatted at 2026-09-27 06:59:41 on dist-test-slave-f892"
server_key: "360c27674c6015ea3625697053d335fd"
server_key_iv: "4ee0e97c756955a9a21a1a4562192b23"
server_key_version: "encryptionkey@0"
I20260927 06:59:41.908886 22006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 06:59:41.912123 22021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:41.913349 22006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 06:59:41.913518 22006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "93dc48f2bdde49f88fa7e9ca325a53a1"
format_stamp: "Formatted at 2026-09-27 06:59:41 on dist-test-slave-f892"
server_key: "360c27674c6015ea3625697053d335fd"
server_key_iv: "4ee0e97c756955a9a21a1a4562192b23"
server_key_version: "encryptionkey@0"
I20260927 06:59:41.913702 22006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:41.918949 22006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:41.922613 22006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:41.922896 22006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:41.939823 22006 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33187
I20260927 06:59:41.939891 22072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33187 every 8 connection(s)
I20260927 06:59:41.941078 22006 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:41.944386 22073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:41.951128 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 22006
I20260927 06:59:41.951303 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:41.951704 388 external_mini_cluster.cc:1428] Setting key 1c260d4d664a3fc01c0f435a79f91fd7
I20260927 06:59:41.953465 22073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1: Bootstrap starting.
I20260927 06:59:41.956410 22073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:41.957691 22073 log.cc:824] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:41.960366 22073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1: No bootstrap required, opened a new log
I20260927 06:59:41.963003 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:41.957377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:50009 (local address 127.0.0.1:33187)
0927 06:59:41.958226 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:41.958239 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:41.958276 (+ 37us) server_negotiation.cc:407] Connection header received
0927 06:59:41.959927 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:41.959948 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:41.960230 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:41.960565 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:41.961508 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:41.961513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:41.961524 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:41.961537 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:41.961587 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:41.961604 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:41.961832 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:41.961974 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 06:59:41.962180 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":383,"threads_started":1}
I20260927 06:59:41.966259 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:41.968030 22073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dc48f2bdde49f88fa7e9ca325a53a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33187 } }
I20260927 06:59:41.968356 22073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:41.968466 22073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93dc48f2bdde49f88fa7e9ca325a53a1, State: Initialized, Role: FOLLOWER
I20260927 06:59:41.969112 22073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dc48f2bdde49f88fa7e9ca325a53a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33187 } }
I20260927 06:59:41.969383 22073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:41.969511 22073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:41.969676 22073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:41.972333 22073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dc48f2bdde49f88fa7e9ca325a53a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33187 } }
I20260927 06:59:41.972626 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:41.970229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:43239 (local address 127.0.0.1:33187)
0927 06:59:41.970365 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:41.970369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:41.970836 (+ 467us) server_negotiation.cc:407] Connection header received
0927 06:59:41.971003 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:41.971008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:41.971066 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:41.971162 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:41.972004 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:41.972009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:41.972013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:41.972031 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:41.972071 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:41.972085 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:41.972207 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:41.972283 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 06:59:41.972343 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 06:59:41.972895 22073 leader_election.cc:302] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93dc48f2bdde49f88fa7e9ca325a53a1; no voters:
I20260927 06:59:41.973459 22073 leader_election.cc:288] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:41.973574 22083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:41.975133 22073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:41.977778 22083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1 [term 1 LEADER]: Becoming Leader. State: Replica: 93dc48f2bdde49f88fa7e9ca325a53a1, State: Running, Role: LEADER
I20260927 06:59:41.978300 22083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dc48f2bdde49f88fa7e9ca325a53a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33187 } }
I20260927 06:59:41.981321 22084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93dc48f2bdde49f88fa7e9ca325a53a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dc48f2bdde49f88fa7e9ca325a53a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33187 } } }
I20260927 06:59:41.981801 22084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:41.982802 22097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:41.982781 22085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93dc48f2bdde49f88fa7e9ca325a53a1. Latest consensus state: current_term: 1 leader_uuid: "93dc48f2bdde49f88fa7e9ca325a53a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dc48f2bdde49f88fa7e9ca325a53a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33187 } } }
I20260927 06:59:41.983137 22085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:41.986974 22097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:41.994359 22097 catalog_manager.cc:1347] Generated new cluster ID: ab52569fe402466ca705ba13eaf69e89
I20260927 06:59:41.994446 22097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:42.013693 22097 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:42.014158 22097 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:42.022403 22097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93dc48f2bdde49f88fa7e9ca325a53a1: Generated new TSK 0
I20260927 06:59:42.023104 22097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:42.032527 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 22006
W20260927 06:59:42.033977 22068 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.18:50009
W20260927 06:59:42.034698 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 22006: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:59:42.034833 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 22006
2026-09-27T06:59:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1827 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:42 dist-test-slave-f892 krb5kdc[22104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:42 dist-test-slave-f892 krb5kdc[22104](info): set up 2 sockets
Sep 27 06:59:42 dist-test-slave-f892 krb5kdc[22104](info): commencing operation
krb5kdc: starting...
W20260927 06:59:43.130436 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.010s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:43 dist-test-slave-f892 krb5kdc[22104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:59:43.352989 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44131
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:43.668674 22117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:43.668936 22117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:43.669023 22117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:43.675683 22117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:43.675776 22117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:43.675835 22117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:43.675906 22117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:43.675945 22117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:43.685222 22117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44131
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:43.686406 22117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:43.687595 22117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:43.698475 22122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:43.698506 22123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:43.698557 22125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:43.699267 22117 server_base.cc:1034] running on GCE node
I20260927 06:59:43.700012 22117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:43.701274 22117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:43.702487 22117 hybrid_clock.cc:648] HybridClock initialized: now 1790492383702439 us; error 38 us; skew 500 ppm
I20260927 06:59:43.702809 22117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:43 dist-test-slave-f892 krb5kdc[22104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:43.705808 22117 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:59:43.707274 22117 webserver.cc:460] Webserver started at http://127.0.97.62:40327/ using document root <none> and password file <none>
I20260927 06:59:43.707921 22117 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:43.708057 22117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:43.708345 22117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:43.711205 22117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "7d8fb02ada8f4aadab7b1776c0cee4c5"
format_stamp: "Formatted at 2026-09-27 06:59:43 on dist-test-slave-f892"
server_key: "ae1c39d167a44d49345638460ca87f7c"
server_key_iv: "364fcdfefa3dc18233a770c418f9da9d"
server_key_version: "encryptionkey@0"
I20260927 06:59:43.712019 22117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "7d8fb02ada8f4aadab7b1776c0cee4c5"
format_stamp: "Formatted at 2026-09-27 06:59:43 on dist-test-slave-f892"
server_key: "ae1c39d167a44d49345638460ca87f7c"
server_key_iv: "364fcdfefa3dc18233a770c418f9da9d"
server_key_version: "encryptionkey@0"
I20260927 06:59:43.716962 22117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 06:59:43.720386 22132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:43.721736 22117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 06:59:43.721913 22117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "7d8fb02ada8f4aadab7b1776c0cee4c5"
format_stamp: "Formatted at 2026-09-27 06:59:43 on dist-test-slave-f892"
server_key: "ae1c39d167a44d49345638460ca87f7c"
server_key_iv: "364fcdfefa3dc18233a770c418f9da9d"
server_key_version: "encryptionkey@0"
I20260927 06:59:43.722179 22117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:43.742445 22117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:43.746665 22117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:43.746941 22117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:43.764514 22117 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44131
I20260927 06:59:43.764549 22183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44131 every 8 connection(s)
I20260927 06:59:43.765741 22117 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:43.769147 22184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:43.772559 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 22117
I20260927 06:59:43.772733 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:43.773171 388 external_mini_cluster.cc:1428] Setting key 843613fb4d8e67631e7c126c26825556
I20260927 06:59:43.781093 22184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5: Bootstrap starting.
I20260927 06:59:43.784653 22184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:43.785907 22184 log.cc:824] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5: Log is configured to *not* fsync() on all Append() calls
Sep 27 06:59:43 dist-test-slave-f892 krb5kdc[22104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:59:43.788549 22184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5: No bootstrap required, opened a new log
I20260927 06:59:43.795080 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:43.774922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54141 (local address 127.0.97.62:44131)
0927 06:59:43.775664 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:43.775675 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:43.775725 (+ 50us) server_negotiation.cc:407] Connection header received
0927 06:59:43.777200 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:43.777219 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:43.777579 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:43.777878 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:43.779550 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:43.780767 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:43.782506 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:43.783150 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:43.783272 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:43.788062 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:43.788069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:43.788082 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:43.788114 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:43.791168 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:43.791905 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:43.791911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:43.791916 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:43.792072 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:43.792518 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:43.792524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:43.792526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:43.792998 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:43.793327 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:43.793773 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 06:59:43.794147 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":358,"threads_started":1}
I20260927 06:59:43.796259 22184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8fb02ada8f4aadab7b1776c0cee4c5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44131 } }
I20260927 06:59:43.796605 22184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:43.796711 22184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d8fb02ada8f4aadab7b1776c0cee4c5, State: Initialized, Role: FOLLOWER
I20260927 06:59:43.797307 22184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8fb02ada8f4aadab7b1776c0cee4c5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44131 } }
I20260927 06:59:43.797586 22184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:43.797724 22184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:43.797899 22184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:43.798628 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:43.800091 22184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8fb02ada8f4aadab7b1776c0cee4c5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44131 } }
I20260927 06:59:43.800519 22184 leader_election.cc:302] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d8fb02ada8f4aadab7b1776c0cee4c5; no voters:
I20260927 06:59:43.801128 22184 leader_election.cc:288] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:43.801234 22190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:43.802757 22190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5 [term 1 LEADER]: Becoming Leader. State: Replica: 7d8fb02ada8f4aadab7b1776c0cee4c5, State: Running, Role: LEADER
I20260927 06:59:43.803367 22184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:43.803372 22190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8fb02ada8f4aadab7b1776c0cee4c5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44131 } }
I20260927 06:59:43.811546 22195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d8fb02ada8f4aadab7b1776c0cee4c5. Latest consensus state: current_term: 1 leader_uuid: "7d8fb02ada8f4aadab7b1776c0cee4c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8fb02ada8f4aadab7b1776c0cee4c5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44131 } } }
I20260927 06:59:43.811898 22195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:43.815687 22193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d8fb02ada8f4aadab7b1776c0cee4c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8fb02ada8f4aadab7b1776c0cee4c5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44131 } } }
I20260927 06:59:43.816032 22193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:43.816911 22206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:43.822384 22206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:43.825100 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:43.803525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41497 (local address 127.0.97.62:44131)
0927 06:59:43.806957 (+ 3432us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:43.806962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:43.806980 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:59:43.807145 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:43.807153 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:43.807246 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:43.807367 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:43.809466 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:43.810655 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:43.812689 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:43.813623 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:43.813768 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:43.817340 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:43.817347 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:43.817352 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:43.817397 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:43.821079 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:43.822068 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:43.822074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:43.822077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:43.822233 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:43.822731 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:43.822738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:43.822746 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:43.823178 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:43.824373 (+ 1195us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:43.824545 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 06:59:43.824717 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3305}
I20260927 06:59:43.831280 22206 catalog_manager.cc:1347] Generated new cluster ID: bb520206af804834952267f3639722eb
I20260927 06:59:43.831380 22206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:43.852815 22206 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:43.854291 22206 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:43.869958 22206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d8fb02ada8f4aadab7b1776c0cee4c5: Generated new TSK 0
I20260927 06:59:43.870769 22206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:43.882771 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 22117
I20260927 06:59:43.884164 22179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:43.973948 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 22117
W20260927 06:59:43.990846 22102 connection.cc:535] client connection to 127.0.97.62:44131 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:44131): Connection reset by peer (error 104)
2026-09-27T06:59:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1939 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:44 dist-test-slave-f892 krb5kdc[22217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:44 dist-test-slave-f892 krb5kdc[22217](info): set up 2 sockets
Sep 27 06:59:44 dist-test-slave-f892 krb5kdc[22217](info): commencing operation
krb5kdc: starting...
W20260927 06:59:45.066555 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:45 dist-test-slave-f892 krb5kdc[22217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:59:45.295686 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34179
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:45.611562 22230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:45.611840 22230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:45.611958 22230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:45.618687 22230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:45.618803 22230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:45.618881 22230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:45.618969 22230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:45.619041 22230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:45.628075 22230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34179
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:45.629287 22230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:45.630537 22230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:45.641919 22235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:45.641945 22236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:45.641880 22238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:45.642782 22230 server_base.cc:1034] running on GCE node
I20260927 06:59:45.643462 22230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:45.644668 22230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:45.645854 22230 hybrid_clock.cc:648] HybridClock initialized: now 1790492385645814 us; error 37 us; skew 500 ppm
I20260927 06:59:45.646265 22230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:45 dist-test-slave-f892 krb5kdc[22217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:45.649328 22230 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:59:45.650753 22230 webserver.cc:460] Webserver started at http://127.0.97.62:36345/ using document root <none> and password file <none>
I20260927 06:59:45.651383 22230 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:45.651508 22230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:45.651793 22230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:45.654639 22230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "de96828cdfc740e4bb8cb8932e52df85"
format_stamp: "Formatted at 2026-09-27 06:59:45 on dist-test-slave-f892"
server_key: "34779ae4013693dc5a583cc07dabeeda"
server_key_iv: "c82b22b5c4c764f767f41bf54f9194ff"
server_key_version: "encryptionkey@0"
I20260927 06:59:45.655362 22230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "de96828cdfc740e4bb8cb8932e52df85"
format_stamp: "Formatted at 2026-09-27 06:59:45 on dist-test-slave-f892"
server_key: "34779ae4013693dc5a583cc07dabeeda"
server_key_iv: "c82b22b5c4c764f767f41bf54f9194ff"
server_key_version: "encryptionkey@0"
I20260927 06:59:45.660037 22230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 06:59:45.663236 22245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:45.664487 22230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 06:59:45.664669 22230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "de96828cdfc740e4bb8cb8932e52df85"
format_stamp: "Formatted at 2026-09-27 06:59:45 on dist-test-slave-f892"
server_key: "34779ae4013693dc5a583cc07dabeeda"
server_key_iv: "c82b22b5c4c764f767f41bf54f9194ff"
server_key_version: "encryptionkey@0"
I20260927 06:59:45.664851 22230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:45.679725 22230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:45.683773 22230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:45.684032 22230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:45.701220 22230 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34179
I20260927 06:59:45.701260 22296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34179 every 8 connection(s)
I20260927 06:59:45.702510 22230 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:45.705813 22297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:45.708717 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 22230
I20260927 06:59:45.708881 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:45.709295 388 external_mini_cluster.cc:1428] Setting key 1e5db0ce2b1cb9f6707216ea5781c4f0
I20260927 06:59:45.718444 22297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85: Bootstrap starting.
I20260927 06:59:45.721909 22297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:45.723229 22297 log.cc:824] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:45.723898 22300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:45.711062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35829 (local address 127.0.97.62:34179)
0927 06:59:45.711818 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:45.711831 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:45.711883 (+ 52us) server_negotiation.cc:407] Connection header received
0927 06:59:45.714164 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:45.714191 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:45.714673 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:45.715086 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:45.715561 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:45.717518 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:45.718945 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:45.719907 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:45.720091 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:45.721140 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:45.721149 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:45.721164 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:45.721798 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:45.721913 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:45.721946 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:45.722128 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:45.722420 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 06:59:45.722583 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":324,"threads_started":1}
I20260927 06:59:45.725967 22297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85: No bootstrap required, opened a new log
I20260927 06:59:45.727370 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:45.733917 22297 raft_consensus.cc:348] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de96828cdfc740e4bb8cb8932e52df85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34179 } }
I20260927 06:59:45.734326 22297 raft_consensus.cc:374] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:45.734448 22297 raft_consensus.cc:729] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de96828cdfc740e4bb8cb8932e52df85, State: Initialized, Role: FOLLOWER
I20260927 06:59:45.735013 22297 consensus_queue.cc:260] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de96828cdfc740e4bb8cb8932e52df85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34179 } }
I20260927 06:59:45.735266 22297 raft_consensus.cc:388] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:45.735430 22297 raft_consensus.cc:482] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:45.735605 22297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:45.737751 22300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:45.731284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46931 (local address 127.0.97.62:34179)
0927 06:59:45.731468 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:45.731477 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:45.731929 (+ 452us) server_negotiation.cc:407] Connection header received
0927 06:59:45.732071 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:45.732074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:45.732131 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:45.732225 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:45.732814 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:45.733537 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:45.735311 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:45.735858 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:45.735946 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:45.737039 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:45.737043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:45.737046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:45.737061 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:45.737108 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:45.737124 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:45.737202 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:45.737392 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 06:59:45.737452 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 06:59:45.738426 22297 raft_consensus.cc:504] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de96828cdfc740e4bb8cb8932e52df85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34179 } }
I20260927 06:59:45.738888 22297 leader_election.cc:302] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de96828cdfc740e4bb8cb8932e52df85; no voters:
I20260927 06:59:45.739409 22297 leader_election.cc:288] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:45.739526 22307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:45.740964 22307 raft_consensus.cc:686] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85 [term 1 LEADER]: Becoming Leader. State: Replica: de96828cdfc740e4bb8cb8932e52df85, State: Running, Role: LEADER
I20260927 06:59:45.741503 22307 consensus_queue.cc:237] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de96828cdfc740e4bb8cb8932e52df85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34179 } }
I20260927 06:59:45.741709 22297 sys_catalog.cc:564] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:45.744271 22309 sys_catalog.cc:455] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de96828cdfc740e4bb8cb8932e52df85. Latest consensus state: current_term: 1 leader_uuid: "de96828cdfc740e4bb8cb8932e52df85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de96828cdfc740e4bb8cb8932e52df85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34179 } } }
I20260927 06:59:45.744233 22308 sys_catalog.cc:455] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de96828cdfc740e4bb8cb8932e52df85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de96828cdfc740e4bb8cb8932e52df85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34179 } } }
I20260927 06:59:45.744691 22308 sys_catalog.cc:458] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:45.745296 22309 sys_catalog.cc:458] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:45.745517 22312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:45.752658 22312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:45.759320 22324 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 06:59:45.759814 22312 catalog_manager.cc:1347] Generated new cluster ID: d1c4ca1986194a7ba1112ab7de89f281
I20260927 06:59:45.759980 22312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:45.789001 22312 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:45.790292 22312 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:45.798802 22312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de96828cdfc740e4bb8cb8932e52df85: Generated new TSK 0
I20260927 06:59:45.799566 22312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:45.807042 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 22230
W20260927 06:59:45.808373 22292 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35829
W20260927 06:59:45.809126 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 22230: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:59:45.809281 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 22230
W20260927 06:59:45.822754 22215 connection.cc:535] client connection to 127.0.97.62:34179 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:34179): Connection reset by peer (error 104)
2026-09-27T06:59:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1831 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:45 dist-test-slave-f892 krb5kdc[22328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:45 dist-test-slave-f892 krb5kdc[22328](info): set up 2 sockets
Sep 27 06:59:45 dist-test-slave-f892 krb5kdc[22328](info): commencing operation
krb5kdc: starting...
W20260927 06:59:46.890342 388 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:47 dist-test-slave-f892 krb5kdc[22328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:59:47.112179 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43499
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:47.423540 22341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:47.423789 22341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:47.423913 22341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:47.430727 22341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:47.430851 22341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:47.430941 22341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:47.431037 22341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:47.431110 22341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:47.440258 22341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43499
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:47.441465 22341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:47.442667 22341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:47.453498 22347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:47.453611 22346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:47.453656 22349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:47.454351 22341 server_base.cc:1034] running on GCE node
I20260927 06:59:47.455080 22341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:47.456357 22341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:47.457574 22341 hybrid_clock.cc:648] HybridClock initialized: now 1790492387457546 us; error 47 us; skew 500 ppm
I20260927 06:59:47.457906 22341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:47 dist-test-slave-f892 krb5kdc[22328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:47.464411 22341 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:59:47.465790 22341 webserver.cc:460] Webserver started at http://127.0.97.62:43635/ using document root <none> and password file <none>
I20260927 06:59:47.466485 22341 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:47.466621 22341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:47.466907 22341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:47.469584 22341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "a119338ddff24a0fa41fa6ba7029a376"
format_stamp: "Formatted at 2026-09-27 06:59:47 on dist-test-slave-f892"
server_key: "1d0649b6f2adfcab2252a6b8eac15db2"
server_key_iv: "3a053eeaeebd23135f3ab09b2585f028"
server_key_version: "encryptionkey@0"
I20260927 06:59:47.470364 22341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "a119338ddff24a0fa41fa6ba7029a376"
format_stamp: "Formatted at 2026-09-27 06:59:47 on dist-test-slave-f892"
server_key: "1d0649b6f2adfcab2252a6b8eac15db2"
server_key_iv: "3a053eeaeebd23135f3ab09b2585f028"
server_key_version: "encryptionkey@0"
I20260927 06:59:47.475075 22341 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 06:59:47.478252 22356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:47.479422 22341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 06:59:47.479592 22341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "a119338ddff24a0fa41fa6ba7029a376"
format_stamp: "Formatted at 2026-09-27 06:59:47 on dist-test-slave-f892"
server_key: "1d0649b6f2adfcab2252a6b8eac15db2"
server_key_iv: "3a053eeaeebd23135f3ab09b2585f028"
server_key_version: "encryptionkey@0"
I20260927 06:59:47.479787 22341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:47.485235 22341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:47.489229 22341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:47.489506 22341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:47.507030 22341 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43499
I20260927 06:59:47.507104 22407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43499 every 8 connection(s)
I20260927 06:59:47.508315 22341 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:47.511404 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 22341
I20260927 06:59:47.511582 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:47.511852 22408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:47.512117 388 external_mini_cluster.cc:1428] Setting key 372c639cd887d68108788c92c0eb7798
I20260927 06:59:47.519722 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:47.513832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57833 (local address 127.0.97.62:43499)
0927 06:59:47.514667 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:47.514681 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:47.514722 (+ 41us) server_negotiation.cc:407] Connection header received
0927 06:59:47.516463 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:47.516483 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:47.516806 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:47.517224 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:47.518260 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:47.518266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:47.518277 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:47.518289 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:47.518343 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:47.518360 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:47.518587 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:47.518731 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 06:59:47.518895 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":395,"threads_started":1}
I20260927 06:59:47.523550 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:47.525302 22408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376: Bootstrap starting.
I20260927 06:59:47.529117 22408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:47.530435 22408 log.cc:824] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:47.531047 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:47.528501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38469 (local address 127.0.97.62:43499)
0927 06:59:47.528714 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:47.528720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:47.529081 (+ 361us) server_negotiation.cc:407] Connection header received
0927 06:59:47.529298 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:47.529304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:47.529357 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:47.529444 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:47.530436 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:47.530440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:47.530443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:47.530457 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:47.530490 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:47.530503 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:47.530626 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:47.530736 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 06:59:47.530777 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 06:59:47.533754 22408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376: No bootstrap required, opened a new log
I20260927 06:59:47.540913 22408 raft_consensus.cc:348] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a119338ddff24a0fa41fa6ba7029a376" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43499 } }
I20260927 06:59:47.541260 22408 raft_consensus.cc:374] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:47.541358 22408 raft_consensus.cc:729] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a119338ddff24a0fa41fa6ba7029a376, State: Initialized, Role: FOLLOWER
I20260927 06:59:47.541870 22408 consensus_queue.cc:260] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a119338ddff24a0fa41fa6ba7029a376" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43499 } }
I20260927 06:59:47.542124 22408 raft_consensus.cc:388] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:47.542261 22408 raft_consensus.cc:482] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:47.542425 22408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:47.544450 22408 raft_consensus.cc:504] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a119338ddff24a0fa41fa6ba7029a376" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43499 } }
I20260927 06:59:47.544946 22408 leader_election.cc:302] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a119338ddff24a0fa41fa6ba7029a376; no voters:
I20260927 06:59:47.545483 22408 leader_election.cc:288] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:47.545601 22418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:47.546283 22418 raft_consensus.cc:686] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376 [term 1 LEADER]: Becoming Leader. State: Replica: a119338ddff24a0fa41fa6ba7029a376, State: Running, Role: LEADER
I20260927 06:59:47.546861 22418 consensus_queue.cc:237] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a119338ddff24a0fa41fa6ba7029a376" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43499 } }
I20260927 06:59:47.547629 22408 sys_catalog.cc:564] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:47.551132 22420 sys_catalog.cc:455] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a119338ddff24a0fa41fa6ba7029a376. Latest consensus state: current_term: 1 leader_uuid: "a119338ddff24a0fa41fa6ba7029a376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a119338ddff24a0fa41fa6ba7029a376" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43499 } } }
I20260927 06:59:47.551578 22420 sys_catalog.cc:458] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:47.552315 22419 sys_catalog.cc:455] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a119338ddff24a0fa41fa6ba7029a376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a119338ddff24a0fa41fa6ba7029a376" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43499 } } }
I20260927 06:59:47.553099 22419 sys_catalog.cc:458] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:47.554968 22431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:47.559876 22431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:47.567850 22431 catalog_manager.cc:1347] Generated new cluster ID: 686a92e2fac14d5f875e989e7607d77e
I20260927 06:59:47.567965 22431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:47.608680 22431 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:47.609071 22431 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:47.621852 22431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a119338ddff24a0fa41fa6ba7029a376: Generated new TSK 0
I20260927 06:59:47.622620 22431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:47.636475 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 22341
W20260927 06:59:47.637501 22403 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57833
W20260927 06:59:47.638191 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 22341: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:59:47.638326 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 22341
2026-09-27T06:59:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1829 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:47 dist-test-slave-f892 krb5kdc[22440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:47 dist-test-slave-f892 krb5kdc[22440](info): set up 2 sockets
Sep 27 06:59:47 dist-test-slave-f892 krb5kdc[22440](info): commencing operation
krb5kdc: starting...
W20260927 06:59:48.722476 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:48 dist-test-slave-f892 krb5kdc[22440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:59:48.955083 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36899
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:49.262694 22453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:49.262939 22453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:49.263088 22453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:49.270308 22453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:49.270403 22453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:49.270500 22453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:49.270613 22453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:49.270671 22453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:49.280375 22453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:49.281558 22453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:49.282819 22453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:49.293835 22458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:49.293887 22459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:49.293921 22461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:49.294725 22453 server_base.cc:1034] running on GCE node
I20260927 06:59:49.295475 22453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:49.296752 22453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:49.297983 22453 hybrid_clock.cc:648] HybridClock initialized: now 1790492389297899 us; error 72 us; skew 500 ppm
I20260927 06:59:49.298370 22453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:49 dist-test-slave-f892 krb5kdc[22440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:49.301259 22453 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:59:49.302644 22453 webserver.cc:460] Webserver started at http://127.0.97.62:34005/ using document root <none> and password file <none>
I20260927 06:59:49.303289 22453 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:49.303436 22453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:49.303726 22453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:49.306558 22453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "90fceae542a34d46af5f5f6c6cc00b8b"
format_stamp: "Formatted at 2026-09-27 06:59:49 on dist-test-slave-f892"
server_key: "185f98fa428deb81f3dde831d5183608"
server_key_iv: "41318fcc9637efaad1fb3f1fa0a1d93f"
server_key_version: "encryptionkey@0"
I20260927 06:59:49.307302 22453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "90fceae542a34d46af5f5f6c6cc00b8b"
format_stamp: "Formatted at 2026-09-27 06:59:49 on dist-test-slave-f892"
server_key: "185f98fa428deb81f3dde831d5183608"
server_key_iv: "41318fcc9637efaad1fb3f1fa0a1d93f"
server_key_version: "encryptionkey@0"
I20260927 06:59:49.312096 22453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 06:59:49.315294 22468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:49.316501 22453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 06:59:49.316697 22453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "90fceae542a34d46af5f5f6c6cc00b8b"
format_stamp: "Formatted at 2026-09-27 06:59:49 on dist-test-slave-f892"
server_key: "185f98fa428deb81f3dde831d5183608"
server_key_iv: "41318fcc9637efaad1fb3f1fa0a1d93f"
server_key_version: "encryptionkey@0"
I20260927 06:59:49.316900 22453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:49.369644 22453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:49.373504 22453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:49.373782 22453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:49.391961 22453 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36899
I20260927 06:59:49.392040 22519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36899 every 8 connection(s)
I20260927 06:59:49.393796 22453 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:49.394986 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 22453
I20260927 06:59:49.395171 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:49.395605 388 external_mini_cluster.cc:1428] Setting key 3275b2d068a7c1abd9f7c21bff321c22
I20260927 06:59:49.398236 22520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 06:59:49 dist-test-slave-f892 krb5kdc[22440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:59:49.408968 22520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b: Bootstrap starting.
I20260927 06:59:49.412973 22520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:49.414327 22520 log.cc:824] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:49.417740 22520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b: No bootstrap required, opened a new log
I20260927 06:59:49.418588 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:49.397310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53519 (local address 127.0.97.62:36899)
0927 06:59:49.398066 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:49.398082 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:49.398132 (+ 50us) server_negotiation.cc:407] Connection header received
0927 06:59:49.399759 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:49.399783 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:49.400127 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:49.400505 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:49.401909 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:49.403504 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:49.404804 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:49.405493 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:49.405702 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:49.409748 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:49.409758 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:49.409776 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:49.409820 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:49.413816 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:49.414649 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:49.414659 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:49.414667 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:49.414912 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:49.415509 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:49.415513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:49.415515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:49.416165 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:49.416439 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:49.416842 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 06:59:49.417251 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":310,"threads_started":1}
I20260927 06:59:49.422219 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:49.426072 22520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fceae542a34d46af5f5f6c6cc00b8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36899 } }
I20260927 06:59:49.426507 22520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:49.426676 22520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90fceae542a34d46af5f5f6c6cc00b8b, State: Initialized, Role: FOLLOWER
I20260927 06:59:49.427325 22520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fceae542a34d46af5f5f6c6cc00b8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36899 } }
I20260927 06:59:49.427629 22520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:49.427778 22520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:49.427968 22520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:49.430806 22520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fceae542a34d46af5f5f6c6cc00b8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36899 } }
I20260927 06:59:49.431329 22520 leader_election.cc:302] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90fceae542a34d46af5f5f6c6cc00b8b; no voters:
I20260927 06:59:49.432044 22520 leader_election.cc:288] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:49.432160 22530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:49.432850 22530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b [term 1 LEADER]: Becoming Leader. State: Replica: 90fceae542a34d46af5f5f6c6cc00b8b, State: Running, Role: LEADER
I20260927 06:59:49.433578 22530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fceae542a34d46af5f5f6c6cc00b8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36899 } }
I20260927 06:59:49.437017 22520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:49.440831 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:49.426392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41003 (local address 127.0.97.62:36899)
0927 06:59:49.426596 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:49.426605 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:49.427030 (+ 425us) server_negotiation.cc:407] Connection header received
0927 06:59:49.427179 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:49.427182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:49.427237 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:49.427316 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:49.429150 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:49.429884 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:49.431631 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:49.432112 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:49.432259 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:49.434903 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:49.434914 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:49.434921 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:49.434966 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:49.438061 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:49.438735 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:49.438745 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:49.438748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:49.438919 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:49.439400 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:49.439404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:49.439406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:49.439670 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:49.439965 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:49.440234 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 06:59:49.440447 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 06:59:49.443395 22532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90fceae542a34d46af5f5f6c6cc00b8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fceae542a34d46af5f5f6c6cc00b8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36899 } } }
I20260927 06:59:49.443420 22531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90fceae542a34d46af5f5f6c6cc00b8b. Latest consensus state: current_term: 1 leader_uuid: "90fceae542a34d46af5f5f6c6cc00b8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fceae542a34d46af5f5f6c6cc00b8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36899 } } }
I20260927 06:59:49.443823 22532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:49.443950 22531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:49.446978 22537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:49.451476 22537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:49.459933 22537 catalog_manager.cc:1347] Generated new cluster ID: 6ed7d503532844478ff8af98c6251dfc
I20260927 06:59:49.460049 22537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:49.468364 22537 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:49.469597 22537 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:49.476930 22537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90fceae542a34d46af5f5f6c6cc00b8b: Generated new TSK 0
I20260927 06:59:49.477636 22537 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:49.483289 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 22453
I20260927 06:59:49.484723 22515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:49.569998 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 22453
2026-09-27T06:59:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1933 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:49 dist-test-slave-f892 krb5kdc[22552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:49 dist-test-slave-f892 krb5kdc[22552](info): set up 2 sockets
Sep 27 06:59:49 dist-test-slave-f892 krb5kdc[22552](info): commencing operation
krb5kdc: starting...
W20260927 06:59:50.642421 388 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:50 dist-test-slave-f892 krb5kdc[22552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:59:50.871215 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36679
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:51.179320 22565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:51.179623 22565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:51.179791 22565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:51.186705 22565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:51.186841 22565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:51.186954 22565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:51.187063 22565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:51.187150 22565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:51.196305 22565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:51.197435 22565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:51.198618 22565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:51.209632 22570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:51.209753 22573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:51.209780 22571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:51.210553 22565 server_base.cc:1034] running on GCE node
I20260927 06:59:51.211251 22565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:51.212476 22565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:51.213703 22565 hybrid_clock.cc:648] HybridClock initialized: now 1790492391213660 us; error 35 us; skew 500 ppm
I20260927 06:59:51.214077 22565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:51 dist-test-slave-f892 krb5kdc[22552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:51.220510 22565 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:59:51.221948 22565 webserver.cc:460] Webserver started at http://127.0.97.62:35311/ using document root <none> and password file <none>
I20260927 06:59:51.222592 22565 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:51.222733 22565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:51.223037 22565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:51.225771 22565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "dd459e657c3a44a686d7993718f0e35c"
format_stamp: "Formatted at 2026-09-27 06:59:51 on dist-test-slave-f892"
server_key: "829be7c14a9e7bb8804072118e306ec8"
server_key_iv: "cbfd42908e2bc697a1fc77d85830abc4"
server_key_version: "encryptionkey@0"
I20260927 06:59:51.226557 22565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "dd459e657c3a44a686d7993718f0e35c"
format_stamp: "Formatted at 2026-09-27 06:59:51 on dist-test-slave-f892"
server_key: "829be7c14a9e7bb8804072118e306ec8"
server_key_iv: "cbfd42908e2bc697a1fc77d85830abc4"
server_key_version: "encryptionkey@0"
I20260927 06:59:51.231180 22565 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 06:59:51.234299 22580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:51.235427 22565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 06:59:51.235600 22565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "dd459e657c3a44a686d7993718f0e35c"
format_stamp: "Formatted at 2026-09-27 06:59:51 on dist-test-slave-f892"
server_key: "829be7c14a9e7bb8804072118e306ec8"
server_key_iv: "cbfd42908e2bc697a1fc77d85830abc4"
server_key_version: "encryptionkey@0"
I20260927 06:59:51.235785 22565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:51.274590 22565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:51.278573 22565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:51.278936 22565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:51.296458 22565 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36679
I20260927 06:59:51.296491 22631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36679 every 8 connection(s)
I20260927 06:59:51.297773 22565 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:51.301242 22632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:51.307281 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 22565
I20260927 06:59:51.307472 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:51.307848 388 external_mini_cluster.cc:1428] Setting key a8b1cdeb60b45192aa6a583ba41a44e2
I20260927 06:59:51.311734 22632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c: Bootstrap starting.
I20260927 06:59:51.314929 22632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:51.316069 22632 log.cc:824] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:51.319022 22632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c: No bootstrap required, opened a new log
I20260927 06:59:51.320936 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:51.309676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60683 (local address 127.0.97.62:36679)
0927 06:59:51.310451 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:51.310461 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:51.310498 (+ 37us) server_negotiation.cc:407] Connection header received
0927 06:59:51.312075 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:51.312094 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:51.312451 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:51.312727 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:51.313332 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:51.314628 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:51.316478 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:51.317121 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:51.317247 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:51.318371 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:51.318380 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:51.318395 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:51.319063 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:51.319143 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:51.319169 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:51.319291 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:51.319517 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 06:59:51.319780 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":392,"threads_started":1}
I20260927 06:59:51.324422 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:51.327571 22632 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd459e657c3a44a686d7993718f0e35c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36679 } }
I20260927 06:59:51.327899 22632 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:51.327996 22632 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd459e657c3a44a686d7993718f0e35c, State: Initialized, Role: FOLLOWER
I20260927 06:59:51.328641 22632 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd459e657c3a44a686d7993718f0e35c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36679 } }
I20260927 06:59:51.328945 22632 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:51.329087 22632 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:51.329270 22632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:51.332190 22632 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd459e657c3a44a686d7993718f0e35c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36679 } }
I20260927 06:59:51.332751 22632 leader_election.cc:302] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd459e657c3a44a686d7993718f0e35c; no voters:
I20260927 06:59:51.333403 22632 leader_election.cc:288] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:51.333524 22642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:51.334175 22642 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c [term 1 LEADER]: Becoming Leader. State: Replica: dd459e657c3a44a686d7993718f0e35c, State: Running, Role: LEADER
I20260927 06:59:51.334792 22642 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd459e657c3a44a686d7993718f0e35c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36679 } }
I20260927 06:59:51.335513 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:51.328496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49017 (local address 127.0.97.62:36679)
0927 06:59:51.328704 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:51.328712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:51.329231 (+ 519us) server_negotiation.cc:407] Connection header received
0927 06:59:51.329409 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:51.329414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:51.329483 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:51.329559 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:51.330159 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:51.330846 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:51.332656 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:51.333217 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:51.333359 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:51.334519 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:51.334525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:51.334531 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:51.334552 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:51.334611 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:51.334631 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:51.334715 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:51.335138 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 06:59:51.335229 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 06:59:51.335887 22632 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:51.337723 22644 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd459e657c3a44a686d7993718f0e35c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd459e657c3a44a686d7993718f0e35c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36679 } } }
I20260927 06:59:51.338076 22644 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:51.338559 22643 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd459e657c3a44a686d7993718f0e35c. Latest consensus state: current_term: 1 leader_uuid: "dd459e657c3a44a686d7993718f0e35c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd459e657c3a44a686d7993718f0e35c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36679 } } }
I20260927 06:59:51.341749 22643 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:51.343055 22649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:51.348744 22649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:51.359089 22649 catalog_manager.cc:1347] Generated new cluster ID: 5bce21dae5aa45a4b6afaf5be0bc25aa
I20260927 06:59:51.359176 22649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:51.376441 22649 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:51.378221 22649 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:51.392403 22649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd459e657c3a44a686d7993718f0e35c: Generated new TSK 0
I20260927 06:59:51.393416 22649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:51.408681 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 22565
W20260927 06:59:51.410405 22627 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60683
W20260927 06:59:51.411116 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 22565: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:59:51.411262 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 22565
2026-09-27T06:59:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1840 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:51 dist-test-slave-f892 krb5kdc[22663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:51 dist-test-slave-f892 krb5kdc[22663](info): set up 2 sockets
Sep 27 06:59:51 dist-test-slave-f892 krb5kdc[22663](info): commencing operation
krb5kdc: starting...
W20260927 06:59:52.486318 388 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:52 dist-test-slave-f892 krb5kdc[22663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:59:52.716998 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35465
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:53.024416 22676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:53.024708 22676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:53.024870 22676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:53.031697 22676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:53.031849 22676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:53.031958 22676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:53.032053 22676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:53.032136 22676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:53.041152 22676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:53.042296 22676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:53.043493 22676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:53.054857 22684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:53.054836 22681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:53.054831 22682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:53.055563 22676 server_base.cc:1034] running on GCE node
I20260927 06:59:53.056252 22676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:53.057585 22676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:53.058799 22676 hybrid_clock.cc:648] HybridClock initialized: now 1790492393058763 us; error 48 us; skew 500 ppm
I20260927 06:59:53.059109 22676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:53 dist-test-slave-f892 krb5kdc[22663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:53.065157 22676 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:59:53.066560 22676 webserver.cc:460] Webserver started at http://127.0.97.62:36503/ using document root <none> and password file <none>
I20260927 06:59:53.067190 22676 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:53.067330 22676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:53.067620 22676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:53.070399 22676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "ffa543a0c42f4206a282220f36870dd7"
format_stamp: "Formatted at 2026-09-27 06:59:53 on dist-test-slave-f892"
server_key: "61f9350c5ccf95535ae6f70392a7274a"
server_key_iv: "d7d31d9fa27f3bacaa32eca6166fb8de"
server_key_version: "encryptionkey@0"
I20260927 06:59:53.071074 22676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "ffa543a0c42f4206a282220f36870dd7"
format_stamp: "Formatted at 2026-09-27 06:59:53 on dist-test-slave-f892"
server_key: "61f9350c5ccf95535ae6f70392a7274a"
server_key_iv: "d7d31d9fa27f3bacaa32eca6166fb8de"
server_key_version: "encryptionkey@0"
I20260927 06:59:53.075819 22676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 06:59:53.078992 22691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:53.080084 22676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 06:59:53.080250 22676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "ffa543a0c42f4206a282220f36870dd7"
format_stamp: "Formatted at 2026-09-27 06:59:53 on dist-test-slave-f892"
server_key: "61f9350c5ccf95535ae6f70392a7274a"
server_key_iv: "d7d31d9fa27f3bacaa32eca6166fb8de"
server_key_version: "encryptionkey@0"
I20260927 06:59:53.080432 22676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:53.085620 22676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:53.089587 22676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:53.089883 22676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:53.106092 22676 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35465
I20260927 06:59:53.106161 22742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35465 every 8 connection(s)
I20260927 06:59:53.107452 22676 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:53.110824 22743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:53.117224 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 22676
I20260927 06:59:53.117380 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:53.117774 388 external_mini_cluster.cc:1428] Setting key 4bd31f2676e5bf7970ccdd29b88d0d60
I20260927 06:59:53.121033 22743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7: Bootstrap starting.
I20260927 06:59:53.124790 22743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:53.126080 22743 log.cc:824] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:53.126353 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:53.119563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52915 (local address 127.0.97.62:35465)
0927 06:59:53.120332 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:53.120346 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:53.120393 (+ 47us) server_negotiation.cc:407] Connection header received
0927 06:59:53.122806 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:53.122832 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:53.123246 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:53.123633 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:53.124546 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:53.124553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:53.124569 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:53.124589 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:53.124655 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:53.124690 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:53.124876 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:53.124999 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 06:59:53.125188 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":310,"threads_started":1}
I20260927 06:59:53.129241 22743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7: No bootstrap required, opened a new log
I20260927 06:59:53.130097 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:53.136363 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:53.133995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42145 (local address 127.0.97.62:35465)
0927 06:59:53.134194 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:53.134207 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:53.134655 (+ 448us) server_negotiation.cc:407] Connection header received
0927 06:59:53.134795 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:53.134798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:53.134840 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:53.134917 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:53.135778 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:53.135782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:53.135784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:53.135796 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:53.135827 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:53.135840 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:53.135930 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:53.136045 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 06:59:53.136088 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 06:59:53.137255 22743 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa543a0c42f4206a282220f36870dd7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35465 } }
I20260927 06:59:53.137694 22743 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:53.137814 22743 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa543a0c42f4206a282220f36870dd7, State: Initialized, Role: FOLLOWER
I20260927 06:59:53.138365 22743 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa543a0c42f4206a282220f36870dd7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35465 } }
I20260927 06:59:53.138592 22743 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:53.138715 22743 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:53.138870 22743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:53.141006 22743 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa543a0c42f4206a282220f36870dd7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35465 } }
I20260927 06:59:53.141381 22743 leader_election.cc:302] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffa543a0c42f4206a282220f36870dd7; no voters:
I20260927 06:59:53.141872 22743 leader_election.cc:288] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:53.141989 22753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:53.143409 22753 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7 [term 1 LEADER]: Becoming Leader. State: Replica: ffa543a0c42f4206a282220f36870dd7, State: Running, Role: LEADER
I20260927 06:59:53.143944 22753 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa543a0c42f4206a282220f36870dd7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35465 } }
I20260927 06:59:53.144496 22743 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:53.146693 22754 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffa543a0c42f4206a282220f36870dd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa543a0c42f4206a282220f36870dd7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35465 } } }
I20260927 06:59:53.147179 22754 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:53.146800 22755 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffa543a0c42f4206a282220f36870dd7. Latest consensus state: current_term: 1 leader_uuid: "ffa543a0c42f4206a282220f36870dd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa543a0c42f4206a282220f36870dd7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35465 } } }
I20260927 06:59:53.148131 22755 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:53.151528 22759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:53.156231 22759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:53.164335 22759 catalog_manager.cc:1347] Generated new cluster ID: 14cb830993e24b7fbf205ea5dbb8db31
I20260927 06:59:53.164443 22759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:53.177431 22759 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:53.177804 22759 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:53.184664 22759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffa543a0c42f4206a282220f36870dd7: Generated new TSK 0
I20260927 06:59:53.185289 22759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:53.190904 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 22676
W20260927 06:59:53.191771 22738 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52915
W20260927 06:59:53.192349 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 22676: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 06:59:53.192514 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 22676
2026-09-27T06:59:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1781 ms)
[----------] 18 tests from AuthTokenIssuingTest (33709 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:53 dist-test-slave-f892 krb5kdc[22774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:53 dist-test-slave-f892 krb5kdc[22774](info): set up 2 sockets
Sep 27 06:59:53 dist-test-slave-f892 krb5kdc[22774](info): commencing operation
krb5kdc: starting...
W20260927 06:59:54.270504 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.013s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:54 dist-test-slave-f892 krb5kdc[22774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:59:54.503825 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35007
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35007,127.0.97.61:36995,127.0.97.60:40333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:54.813014 22787 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 06:59:54.813377 22787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:54.813503 22787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:54.813633 22787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:54.820721 22787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:54.820829 22787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:54.820969 22787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:54.821056 22787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:54.821146 22787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:54.830464 22787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35007,127.0.97.61:36995,127.0.97.60:40333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:54.831677 22787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:54.832947 22787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:54.844182 22792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:54.844234 22795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:54.844208 22793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:54.845259 22787 server_base.cc:1034] running on GCE node
I20260927 06:59:54.845955 22787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:54.847297 22787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:54.848513 22787 hybrid_clock.cc:648] HybridClock initialized: now 1790492394848459 us; error 24 us; skew 500 ppm
I20260927 06:59:54.848847 22787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:54 dist-test-slave-f892 krb5kdc[22774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:54.851884 22787 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:59:54.853161 22787 webserver.cc:460] Webserver started at http://127.0.97.62:35383/ using document root <none> and password file <none>
I20260927 06:59:54.853812 22787 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:54.853938 22787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:54.854267 22787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:54.857023 22787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "fd2576456e374554ad8d8c3b5db476e5"
format_stamp: "Formatted at 2026-09-27 06:59:54 on dist-test-slave-f892"
server_key: "c6f0a2886af06163d670121b7ba3b6eb"
server_key_iv: "e9d82a3a21951528e7c2978032ed13c9"
server_key_version: "encryptionkey@0"
I20260927 06:59:54.857795 22787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "fd2576456e374554ad8d8c3b5db476e5"
format_stamp: "Formatted at 2026-09-27 06:59:54 on dist-test-slave-f892"
server_key: "c6f0a2886af06163d670121b7ba3b6eb"
server_key_iv: "e9d82a3a21951528e7c2978032ed13c9"
server_key_version: "encryptionkey@0"
I20260927 06:59:54.862552 22787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 06:59:54.865609 22802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:54.866866 22787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 06:59:54.867048 22787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "fd2576456e374554ad8d8c3b5db476e5"
format_stamp: "Formatted at 2026-09-27 06:59:54 on dist-test-slave-f892"
server_key: "c6f0a2886af06163d670121b7ba3b6eb"
server_key_iv: "e9d82a3a21951528e7c2978032ed13c9"
server_key_version: "encryptionkey@0"
I20260927 06:59:54.867245 22787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:54.896255 22787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:54.901646 22787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:54.901924 22787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:54.920393 22787 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35007
I20260927 06:59:54.920413 22853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35007 every 8 connection(s)
I20260927 06:59:54.921640 22787 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:54.924486 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 22787
I20260927 06:59:54.924733 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:54.925511 388 external_mini_cluster.cc:1428] Setting key ecda88a240da4b49fc5a383151899cc1
I20260927 06:59:54.925693 22854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:54.935153 22854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 06:59:54 dist-test-slave-f892 krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260927 06:59:54.956455 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:54.939295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57990 (local address 127.0.97.62:35007)
0927 06:59:54.939911 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:54.939923 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:54.939977 (+ 54us) server_negotiation.cc:407] Connection header received
0927 06:59:54.941163 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:54.941173 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:54.941435 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:54.941632 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:54.942732 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:54.944632 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:54.945890 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:54.946907 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:54.947117 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:54.950151 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:54.950164 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:54.950179 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:54.950216 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:54.952753 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:54.953519 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:54.953525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:54.953527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:54.953659 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:54.954114 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:54.954117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:54.954119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:54.954535 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:54.954688 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:54.955305 (+ 617us) server_negotiation.cc:299] Negotiation successful
0927 06:59:54.955544 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":430,"thread_start_us":282,"threads_started":1}
I20260927 06:59:54.956491 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:54.938157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35007 (local address 127.0.0.1:57990)
0927 06:59:54.938883 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:54.938935 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:54.940825 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:54.941672 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:54.941676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:54.942203 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:54.942534 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:54.942550 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:54.944776 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:54.944779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:54.945761 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:54.945771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:54.947121 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:54.947126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:54.947265 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:54.947940 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:54.947959 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:54.949900 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:54.952856 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:54.952864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:54.952872 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:54.953374 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:54.953773 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:54.953780 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:54.953786 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:54.953982 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:54.954681 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:54.954692 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:54.954934 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 06:59:54.955191 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 06:59:54.955749 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":65,"thread_start_us":312,"threads_started":1}
I20260927 06:59:54.959138 22854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } has no permanent_uuid. Determining permanent_uuid...
I20260927 06:59:54.960314 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:54.959816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36995 (local address 127.0.0.1:60212)
0927 06:59:54.960030 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:54.960155 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260927 06:59:54.960589 22804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36995: connect: Connection refused (error 111)
I20260927 06:59:54.962184 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:54.961863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36995 (local address 127.0.0.1:60228)
0927 06:59:54.961979 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:54.962093 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 06:59:54.962452 22854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36995: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36995: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 06:59:54.981281 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36995
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35007,127.0.97.61:36995,127.0.97.60:40333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:59:55.011863 22854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } attempt: 1
I20260927 06:59:55.012972 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.012565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36995 (local address 127.0.0.1:60244)
0927 06:59:55.012748 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.012834 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 06:59:55.013888 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.013566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36995 (local address 127.0.0.1:60256)
0927 06:59:55.013716 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.013776 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 06:59:55.014076 22854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36995: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36995: connect: Connection refused (error 111)
I20260927 06:59:55.082293 22854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } attempt: 2
I20260927 06:59:55.083542 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.083064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36995 (local address 127.0.0.1:60268)
0927 06:59:55.083308 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.083393 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 06:59:55.084590 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.084247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36995 (local address 127.0.0.1:60274)
0927 06:59:55.084368 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.084469 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 06:59:55.084839 22854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36995: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36995: connect: Connection refused (error 111)
I20260927 06:59:55.176134 22854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } attempt: 3
I20260927 06:59:55.177311 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.176898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36995 (local address 127.0.0.1:60278)
0927 06:59:55.177089 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.177178 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 06:59:55.178303 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.177930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36995 (local address 127.0.0.1:60284)
0927 06:59:55.178110 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.178175 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 06:59:55.178460 22854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36995: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36995: connect: Connection refused (error 111)
W20260927 06:59:55.286974 22861 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 06:59:55.287308 22861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:55.287434 22861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:55.287588 22861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:55.294459 22861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:55.294590 22861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:55.294704 22861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:55.294778 22861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:55.294875 22861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:55.303887 22861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35007,127.0.97.61:36995,127.0.97.60:40333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:55.305078 22861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:55.306341 22861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:55.317153 22866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:55.317152 22869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:55.317186 22867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:55.318073 22861 server_base.cc:1034] running on GCE node
I20260927 06:59:55.318715 22861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:55.319854 22861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:55.321063 22861 hybrid_clock.cc:648] HybridClock initialized: now 1790492395320999 us; error 49 us; skew 500 ppm
I20260927 06:59:55.321405 22861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 06:59:55.321664 22854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } attempt: 4
I20260927 06:59:55.322882 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.322254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36995 (local address 127.0.0.1:60288)
0927 06:59:55.322625 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.322718 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260927 06:59:55.323952 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.323521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36995 (local address 127.0.0.1:60298)
0927 06:59:55.323685 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.323789 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 06:59:55.324169 22854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36995: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36995: connect: Connection refused (error 111)
Sep 27 06:59:55 dist-test-slave-f892 krb5kdc[22774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:55.324867 22861 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 06:59:55.326184 22861 webserver.cc:460] Webserver started at http://127.0.97.61:41181/ using document root <none> and password file <none>
I20260927 06:59:55.326834 22861 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:55.326949 22861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:55.327255 22861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:55.329873 22861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-1/data/instance:
uuid: "36250d70cf8a438aa765885031a4da4d"
format_stamp: "Formatted at 2026-09-27 06:59:55 on dist-test-slave-f892"
server_key: "6bc678e1040404401fc83594c8073be5"
server_key_iv: "7e1550017e6e9f67433e6c2d55e406e8"
server_key_version: "encryptionkey@0"
I20260927 06:59:55.330685 22861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-1/wal/instance:
uuid: "36250d70cf8a438aa765885031a4da4d"
format_stamp: "Formatted at 2026-09-27 06:59:55 on dist-test-slave-f892"
server_key: "6bc678e1040404401fc83594c8073be5"
server_key_iv: "7e1550017e6e9f67433e6c2d55e406e8"
server_key_version: "encryptionkey@0"
I20260927 06:59:55.335302 22861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 06:59:55.338352 22876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:55.339515 22861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 06:59:55.339671 22861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-1/wal
uuid: "36250d70cf8a438aa765885031a4da4d"
format_stamp: "Formatted at 2026-09-27 06:59:55 on dist-test-slave-f892"
server_key: "6bc678e1040404401fc83594c8073be5"
server_key_iv: "7e1550017e6e9f67433e6c2d55e406e8"
server_key_version: "encryptionkey@0"
I20260927 06:59:55.339860 22861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:55.356099 22861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:55.359584 22861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:55.359830 22861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:55.377887 22861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36995
I20260927 06:59:55.377905 22927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36995 every 8 connection(s)
I20260927 06:59:55.379153 22861 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-1/data/info.pb
I20260927 06:59:55.382102 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 22861
I20260927 06:59:55.382296 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-1/wal/instance
I20260927 06:59:55.382640 22928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:55.382752 388 external_mini_cluster.cc:1428] Setting key 41ec52cb2e2e2e6a35e21fbee22d11cf
I20260927 06:59:55.388722 22928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 06:59:55 dist-test-slave-f892 krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260927 06:59:55.409425 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.391300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.0.97.62:35007)
0927 06:59:55.391527 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:55.391535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:55.392496 (+ 961us) server_negotiation.cc:407] Connection header received
0927 06:59:55.394390 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:55.394399 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:55.394489 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:55.394609 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:55.395967 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.397118 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.398688 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.399451 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.399595 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:55.403154 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:55.403164 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:55.403172 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:55.403219 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:55.406504 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:55.407179 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:55.407184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:55.407186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:55.407305 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:55.407747 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:55.407752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:55.407755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:55.408038 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:55.408205 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:55.408853 (+ 648us) server_negotiation.cc:299] Negotiation successful
0927 06:59:55.409091 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 06:59:55.410430 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.391601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35007 (local address 127.0.0.1:57994)
0927 06:59:55.392327 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.392376 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:55.394144 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:55.394791 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:55.394799 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:55.395269 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:55.395791 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:55.395807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.397275 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.397278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:55.398528 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:55.398539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.399654 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.399659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:55.399839 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:55.400522 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:55.400543 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:55.402916 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:55.406633 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:55.406639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:55.406652 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:55.407056 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:55.407393 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:55.407396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:55.407398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:55.407610 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:55.408169 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:55.408177 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:55.408515 (+ 338us) client_negotiation.cc:769] Sending connection context
0927 06:59:55.408762 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 06:59:55.409260 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":313,"threads_started":1}
I20260927 06:59:55.412175 22928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 06:59:55 dist-test-slave-f892 krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 06:59:55.428586 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.412961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36995 (local address 127.0.0.1:60308)
0927 06:59:55.413173 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.413197 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:55.413326 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:55.414153 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:55.414156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:55.414486 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:55.414613 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:55.414632 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.416586 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.416594 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:55.417761 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:55.417772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.418630 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.418642 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:55.418846 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:55.419447 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:55.419474 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:55.422435 (+ 2961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:55.426131 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:55.426137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:55.426141 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:55.426503 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:55.426868 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:55.426872 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:55.426876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:55.427017 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:55.427617 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:55.427627 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:55.427768 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 06:59:55.427906 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 06:59:55.428204 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 06:59:55.428695 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.413015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60308 (local address 127.0.97.61:36995)
0927 06:59:55.413634 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:55.413644 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:55.413675 (+ 31us) server_negotiation.cc:407] Connection header received
0927 06:59:55.413733 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:55.413744 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:55.413952 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:55.414123 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:55.414784 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.416428 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.417882 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.418452 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.418554 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:55.422658 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:55.422676 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:55.422710 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:55.422750 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:55.425929 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:55.426631 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:55.426639 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:55.426642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:55.426758 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:55.427116 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:55.427123 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:55.427126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:55.427460 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:55.427675 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:55.428065 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 06:59:55.428321 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":422,"thread_start_us":276,"threads_started":1}
I20260927 06:59:55.431437 22928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } has no permanent_uuid. Determining permanent_uuid...
I20260927 06:59:55.432665 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.432241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:46836)
0927 06:59:55.432415 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.432526 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 06:59:55.432924 22880 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
I20260927 06:59:55.434487 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.434175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:46850)
0927 06:59:55.434320 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.434389 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 06:59:55.434717 22928 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40333: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 06:59:55.440747 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40333
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35007,127.0.97.61:36995,127.0.97.60:40333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:59:55.484081 22928 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } attempt: 1
I20260927 06:59:55.485177 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.484756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:46858)
0927 06:59:55.484942 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.485034 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 06:59:55.486193 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.485805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:46860)
0927 06:59:55.485955 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.486092 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 06:59:55.486402 22928 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40333: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
I20260927 06:59:55.554642 22928 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } attempt: 2
I20260927 06:59:55.555711 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.555288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:46868)
0927 06:59:55.555474 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.555570 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 06:59:55.556684 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.556368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:46880)
0927 06:59:55.556518 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.556580 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 06:59:55.556915 22928 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40333: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
I20260927 06:59:55.623436 22854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } attempt: 5
Sep 27 06:59:55 dist-test-slave-f892 krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 06:59:55.636323 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.624411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.0.97.61:36995)
0927 06:59:55.624640 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:55.624645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:55.624663 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:59:55.624716 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:55.624720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:55.624786 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:55.624886 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:55.625663 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.626447 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.627574 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.627982 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.628108 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:55.631371 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:55.631375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:55.631378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:55.631406 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:55.633886 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:55.634552 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:55.634560 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:55.634564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:55.634696 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:55.635047 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:55.635053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:55.635056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:55.635312 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:55.635462 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:55.635835 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 06:59:55.636041 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 06:59:55.636324 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.624159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36995 (local address 127.0.0.1:60318)
0927 06:59:55.624349 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.624377 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:55.624533 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:55.624933 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:55.624937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:55.625340 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:55.625518 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:55.625526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.626594 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.626605 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:55.627411 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:55.627421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.628120 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.628126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:55.628322 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:55.628964 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:55.628986 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:55.631205 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:55.634065 (+ 2860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:55.634071 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:55.634075 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:55.634427 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:55.634802 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:55.634808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:55.634811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:55.634946 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:55.635426 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:55.635431 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:55.635589 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 06:59:55.635728 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 06:59:55.635978 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 06:59:55.637657 22854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } has no permanent_uuid. Determining permanent_uuid...
I20260927 06:59:55.638686 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.638318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:46896)
0927 06:59:55.638466 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.638545 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 06:59:55.639609 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.639288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:46904)
0927 06:59:55.639441 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.639512 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 06:59:55.639811 22854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40333: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
I20260927 06:59:55.648118 22928 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } attempt: 3
I20260927 06:59:55.649214 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.648674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:46908)
0927 06:59:55.648895 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.649027 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 06:59:55.650380 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.649947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:46924)
0927 06:59:55.650216 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.650285 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
W20260927 06:59:55.650643 22928 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40333: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
I20260927 06:59:55.691052 22854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } attempt: 1
I20260927 06:59:55.692281 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.691721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:46930)
0927 06:59:55.691984 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.692088 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 06:59:55.693192 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.692876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:46938)
0927 06:59:55.693020 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.693092 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 06:59:55.693400 22854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40333: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
W20260927 06:59:55.755599 22935 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 06:59:55.755936 22935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:55.756055 22935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:55.756189 22935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 06:59:55.761583 22854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } attempt: 2
I20260927 06:59:55.762668 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.762226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:46948)
0927 06:59:55.762427 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.762525 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 06:59:55.763247 22935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:55.763373 22935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:55.763494 22935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:55.763573 22935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:55.763650 22935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:55.763852 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.763336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:46954)
0927 06:59:55.763619 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.763715 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260927 06:59:55.764074 22854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40333: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
I20260927 06:59:55.773346 22935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35007,127.0.97.61:36995,127.0.97.60:40333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:55.774670 22935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:55.775923 22935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:55.787076 22940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:55.787029 22941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:55.787108 22943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:55.787858 22935 server_base.cc:1034] running on GCE node
I20260927 06:59:55.788532 22935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:55.789724 22935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:55.791082 22935 hybrid_clock.cc:648] HybridClock initialized: now 1790492395791006 us; error 68 us; skew 500 ppm
I20260927 06:59:55.791409 22935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:55 dist-test-slave-f892 krb5kdc[22774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:55.793874 22928 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } attempt: 4
I20260927 06:59:55.794371 22935 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 06:59:55.795013 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.794485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:46964)
0927 06:59:55.794739 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.794834 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 06:59:55.796043 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.795626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:46970)
0927 06:59:55.795793 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.795909 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 06:59:55.796029 22935 webserver.cc:460] Webserver started at http://127.0.97.60:45853/ using document root <none> and password file <none>
W20260927 06:59:55.796239 22928 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40333: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
I20260927 06:59:55.796728 22935 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:55.796808 22935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:55.797051 22935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:55.799721 22935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-2/data/instance:
uuid: "24c91b0c14e04676b807aba7524f3813"
format_stamp: "Formatted at 2026-09-27 06:59:55 on dist-test-slave-f892"
server_key: "739c4375f211337e26cfe5b5c22271dd"
server_key_iv: "6ec8c696b16fbb0dfab0a96dc7413c80"
server_key_version: "encryptionkey@0"
I20260927 06:59:55.800370 22935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-2/wal/instance:
uuid: "24c91b0c14e04676b807aba7524f3813"
format_stamp: "Formatted at 2026-09-27 06:59:55 on dist-test-slave-f892"
server_key: "739c4375f211337e26cfe5b5c22271dd"
server_key_iv: "6ec8c696b16fbb0dfab0a96dc7413c80"
server_key_version: "encryptionkey@0"
I20260927 06:59:55.804925 22935 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 06:59:55.808012 22950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:55.809180 22935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 06:59:55.809298 22935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-2/wal
uuid: "24c91b0c14e04676b807aba7524f3813"
format_stamp: "Formatted at 2026-09-27 06:59:55 on dist-test-slave-f892"
server_key: "739c4375f211337e26cfe5b5c22271dd"
server_key_iv: "6ec8c696b16fbb0dfab0a96dc7413c80"
server_key_version: "encryptionkey@0"
I20260927 06:59:55.809464 22935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:55.865020 22935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:55.868695 22935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:55.868922 22935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:55.870282 22854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } attempt: 3
I20260927 06:59:55.871402 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.870900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:46982)
0927 06:59:55.871112 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.871191 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 06:59:55.872440 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.872088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:46992)
0927 06:59:55.872272 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.872340 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260927 06:59:55.872711 22854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40333: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40333: connect: Connection refused (error 111)
I20260927 06:59:55.887034 22935 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40333
I20260927 06:59:55.887066 23001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40333 every 8 connection(s)
I20260927 06:59:55.888248 22935 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-2/data/info.pb
I20260927 06:59:55.891804 23002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:55.893230 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 22935
I20260927 06:59:55.893409 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492047466241-388-0/minicluster-data/master-2/wal/instance
I20260927 06:59:55.893934 388 external_mini_cluster.cc:1428] Setting key 59b6695fd83b19540ce5cf9fe8085bf7
I20260927 06:59:55.901147 23002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 06:59:55 dist-test-slave-f892 krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 06:59:55 dist-test-slave-f892 krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:59:55.916652 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.895646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.0.97.60:40333)
0927 06:59:55.896405 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:55.896417 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:55.896462 (+ 45us) server_negotiation.cc:407] Connection header received
0927 06:59:55.898005 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:55.898082 (+ 77us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:55.898429 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:55.898732 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:55.900428 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.901642 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.903406 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.904521 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.904718 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:55.908989 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:55.908999 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:55.909013 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:55.909063 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:55.912293 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:55.913047 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:55.913057 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:55.913068 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:55.913302 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:55.913694 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:55.913701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:55.913705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:55.914402 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:55.914707 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:55.915043 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 06:59:55.915430 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":358,"threads_started":1}
I20260927 06:59:55.918418 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.903685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58004 (local address 127.0.97.62:35007)
0927 06:59:55.903900 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:55.903908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:55.904406 (+ 498us) server_negotiation.cc:407] Connection header received
0927 06:59:55.904610 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:55.904618 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:55.904686 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:55.904790 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:55.905995 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.906697 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.908482 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.908977 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.909114 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:55.912460 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:55.912468 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:55.912476 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:55.912515 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:55.915789 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:55.916429 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:55.916435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:55.916438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:55.916592 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:55.916964 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:55.916969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:55.916972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:55.917220 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:55.917365 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:55.917798 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 06:59:55.918042 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 06:59:55.918503 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.903617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35007 (local address 127.0.0.1:58004)
0927 06:59:55.904275 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.904315 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:55.904484 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:55.904855 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:55.904862 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:55.905614 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:55.905846 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:55.905855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.906867 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.906871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:55.908322 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:55.908338 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.909175 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.909180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:55.909316 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:55.909945 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:55.909970 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:55.912253 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:55.915915 (+ 3662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:55.915924 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:55.915928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:55.916307 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:55.916707 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:55.916710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:55.916712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:55.916866 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:55.917354 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:55.917361 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:55.917526 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 06:59:55.917692 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 06:59:55.918026 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":353,"threads_started":1}
I20260927 06:59:55.920143 23002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 06:59:55 dist-test-slave-f892 krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 27 06:59:55 dist-test-slave-f892 krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:59:55.933879 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.920964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36995 (local address 127.0.0.1:60324)
0927 06:59:55.921201 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.921216 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:55.921305 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:55.921627 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:55.921632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:55.922035 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:55.922205 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:55.922214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.923300 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.923305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:55.924140 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:55.924150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.924818 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.924823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:55.925021 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:55.925631 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:55.925908 (+ 277us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:55.927909 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:55.931681 (+ 3772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:55.931687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:55.931691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:55.932038 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:55.932400 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:55.932405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:55.932408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:55.932540 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:55.932994 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:55.932998 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:55.933167 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 06:59:55.933300 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 06:59:55.933553 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260927 06:59:55.933933 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.920987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.0.97.61:36995)
0927 06:59:55.921176 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:55.921184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:55.921264 (+ 80us) server_negotiation.cc:407] Connection header received
0927 06:59:55.921411 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:55.921417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:55.921497 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:55.921602 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:55.922346 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.923162 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.924283 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.924669 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.924790 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:55.928080 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:55.928086 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:55.928092 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:55.928127 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:55.931581 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:55.932149 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:55.932155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:55.932159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:55.932291 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:55.932642 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:55.932648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:55.932651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:55.932887 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:55.933055 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:55.933388 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 06:59:55.933584 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 06:59:55.934828 23002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } has no permanent_uuid. Determining permanent_uuid...
I20260927 06:59:55.935248 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.919589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.0.97.62:35007)
0927 06:59:55.919738 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:55.919742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:55.919801 (+ 59us) server_negotiation.cc:407] Connection header received
0927 06:59:55.920013 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:55.920018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:55.920089 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:55.920186 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:55.921657 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.922622 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.924343 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.924972 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.925083 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:55.930065 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:55.930074 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:55.930080 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:55.930127 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:55.932797 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:55.933420 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:55.933431 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:55.933436 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:55.933556 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:55.933900 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:55.933905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:55.933909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:55.934305 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:55.934541 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:55.934815 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 06:59:55.934987 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 06:59:55 dist-test-slave-f892 krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 06:59:55 dist-test-slave-f892 krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 06:59:55.948968 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.935603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:47010)
0927 06:59:55.935757 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.935774 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:55.935879 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:55.936265 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:55.936269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:55.936597 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:55.936751 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:55.936761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.938162 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.938170 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:55.939432 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:55.939445 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.940397 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.940402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:55.940546 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:55.941021 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:55.941035 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:55.943198 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:55.946736 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:55.946743 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:55.946747 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:55.947074 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:55.947433 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:55.947437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:55.947440 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:55.947562 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:55.948027 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:55.948031 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:55.948163 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 06:59:55.948293 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 06:59:55.948569 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 06:59:55.949038 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.935627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47010 (local address 127.0.97.60:40333)
0927 06:59:55.935775 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:55.935780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:55.935831 (+ 51us) server_negotiation.cc:407] Connection header received
0927 06:59:55.936000 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:55.936005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:55.936063 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:55.936154 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:55.936909 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.937998 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.939573 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.940245 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.940364 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:55.943406 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:55.943414 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:55.943418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:55.943455 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:55.946620 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:55.947180 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:55.947187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:55.947190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:55.947316 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:55.947670 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:55.947677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:55.947680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:55.947895 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:55.948061 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:55.948374 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 06:59:55.948709 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":75}
I20260927 06:59:55.951576 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.936724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.0.97.61:36995)
0927 06:59:55.936870 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:55.936874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:55.936887 (+ 13us) server_negotiation.cc:407] Connection header received
0927 06:59:55.937008 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:55.937011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:55.937055 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:55.937125 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:55.938526 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.939550 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.941114 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:55.941746 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:55.941873 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:55.946429 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:55.946433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:55.946436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:55.946462 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:55.948861 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:55.949589 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:55.949598 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:55.949601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:55.949707 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:55.950282 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:55.950287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:55.950289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:55.950578 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:55.950881 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:55.951064 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 06:59:55.951271 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 06:59:55.954842 23002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813: Bootstrap starting.
I20260927 06:59:55.956020 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:55.958559 23002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:55.959609 23002 log.cc:824] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:55.959267 22829 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fd2576456e374554ad8d8c3b5db476e5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58014
I20260927 06:59:55.961916 23002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813: No bootstrap required, opened a new log
I20260927 06:59:55.968575 23002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } } peers { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } } peers { permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } }
I20260927 06:59:55.968858 23002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c91b0c14e04676b807aba7524f3813, State: Initialized, Role: FOLLOWER
I20260927 06:59:55.969259 23002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } } peers { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } } peers { permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } }
I20260927 06:59:55.970156 23008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } } peers { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } } peers { permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } } }
I20260927 06:59:55.970522 23008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:59:55.971076 23002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 06:59:55.979864 23019 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 06:59:55.979979 23019 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 06:59:55.978035 22935 master_runner.cc:186] Error getting master registration for 127.0.97.62:35007: OK, instance_id { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" instance_seqno: 1790492394915483 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 06:59:55.986073 22935 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 06:59:56.049924 22854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } attempt: 4
Sep 27 06:59:56 dist-test-slave-f892 krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 06:59:56.065934 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:56.050738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47020 (local address 127.0.97.60:40333)
0927 06:59:56.050975 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:56.050981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:56.051005 (+ 24us) server_negotiation.cc:407] Connection header received
0927 06:59:56.051155 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:56.051164 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:56.051240 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:56.051342 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:56.052267 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:56.053409 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:56.055200 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:56.055863 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:56.055985 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:56.059721 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:56.059727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:56.059733 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:56.059774 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:56.063324 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:56.064042 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:56.064048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:56.064052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:56.064186 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:56.064562 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:56.064568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:56.064571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:56.064878 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:56.065042 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:56.065384 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 06:59:56.065586 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 06:59:56.065940 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:56.050666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:47020)
0927 06:59:56.050883 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:56.050904 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:56.051028 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:56.051401 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:56.051406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:56.051924 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:56.052127 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:56.052139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:56.053569 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:56.053574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:56.055072 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:56.055088 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:56.056024 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:56.056031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:56.056224 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:56.056882 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:56.056914 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:56.059538 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:56.063504 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:56.063510 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:56.063516 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:56.063914 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:56.064303 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:56.064308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:56.064311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:56.064456 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:56.064995 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:56.065000 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:56.065146 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 06:59:56.065280 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 06:59:56.065547 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 06:59:56.071727 22854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5: Bootstrap starting.
I20260927 06:59:56.075632 22854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:56.076900 22854 log.cc:824] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:56.080052 22854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5: No bootstrap required, opened a new log
I20260927 06:59:56.087399 22854 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } } peers { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } } peers { permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } }
I20260927 06:59:56.087776 22854 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd2576456e374554ad8d8c3b5db476e5, State: Initialized, Role: FOLLOWER
I20260927 06:59:56.088233 22854 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } } peers { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } } peers { permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } }
I20260927 06:59:56.089181 23022 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } } peers { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } } peers { permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } } }
I20260927 06:59:56.089521 23022 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:59:56.090278 22854 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:56.090387 22829 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 06:59:56.090617 22829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:56.093550 22829 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } } peers { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } } peers { permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } }
I20260927 06:59:56.095455 22928 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } attempt: 5
I20260927 06:59:56.095726 22977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd2576456e374554ad8d8c3b5db476e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "24c91b0c14e04676b807aba7524f3813"
I20260927 06:59:56.096165 22977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:56.099476 22977 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd2576456e374554ad8d8c3b5db476e5 in term 1.
I20260927 06:59:56.100096 22806 leader_election.cc:302] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c91b0c14e04676b807aba7524f3813, fd2576456e374554ad8d8c3b5db476e5; no voters:
I20260927 06:59:56.100494 23022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:56.103709 23022 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [term 1 LEADER]: Becoming Leader. State: Replica: fd2576456e374554ad8d8c3b5db476e5, State: Running, Role: LEADER
I20260927 06:59:56.104424 23022 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } } peers { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } } peers { permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } }
Sep 27 06:59:56 dist-test-slave-f892 krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 06:59:56.103677 22829 leader_election.cc:288] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [CANDIDATE]: Term 1 election: Requested vote from peers 36250d70cf8a438aa765885031a4da4d (127.0.97.61:36995), 24c91b0c14e04676b807aba7524f3813 (127.0.97.60:40333)
I20260927 06:59:56.114585 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:56.095972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40333 (local address 127.0.0.1:47026)
0927 06:59:56.096308 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:56.096326 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:56.096434 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:56.097270 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:56.097278 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:56.097755 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:56.097944 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:56.097956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:56.099455 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:56.099463 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:56.100902 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:56.100919 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:56.102530 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:56.102539 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:56.102818 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:56.103577 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:56.103603 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:56.106586 (+ 2983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:56.111364 (+ 4778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:56.111372 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:56.111379 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:56.111961 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:56.112446 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:56.112453 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:56.112460 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:56.112641 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:56.113331 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:56.113338 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:56.113527 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 06:59:56.113691 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 06:59:56.114042 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260927 06:59:56.115919 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:56.096640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.0.97.60:40333)
0927 06:59:56.096822 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:56.096828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:56.096847 (+ 19us) server_negotiation.cc:407] Connection header received
0927 06:59:56.096894 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:56.096899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:56.096996 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:56.097091 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:56.098173 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:56.099283 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:56.101229 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:56.102109 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:56.102259 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:56.106792 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:56.106801 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:56.106807 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:56.106852 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:56.111182 (+ 4330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:56.112106 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:56.112114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:56.112120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:56.112284 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:56.112775 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:56.112782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:56.112788 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:56.113129 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:56.113265 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:56.115150 (+ 1885us) server_negotiation.cc:299] Negotiation successful
0927 06:59:56.115413 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 06:59:56.121110 22956 tablet.cc:2369] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:59:56.122991 22928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d: Bootstrap starting.
I20260927 06:59:56.133790 23028 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd2576456e374554ad8d8c3b5db476e5. Latest consensus state: current_term: 1 leader_uuid: "fd2576456e374554ad8d8c3b5db476e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } } peers { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } } peers { permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } } }
I20260927 06:59:56.134194 23028 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:56.135913 22928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:56.140637 22928 log.cc:824] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:56.142585 23041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:56.152799 22928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d: No bootstrap required, opened a new log
W20260927 06:59:56.155257 22808 tablet.cc:2369] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:59:56.165584 23041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:56.174548 22928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } } peers { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } } peers { permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } }
I20260927 06:59:56.175150 22928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36250d70cf8a438aa765885031a4da4d, State: Initialized, Role: FOLLOWER
I20260927 06:59:56.175221 23045 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 06:59:56.175879 22928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } } peers { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } } peers { permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } }
I20260927 06:59:56.186846 22977 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813 [term 1 FOLLOWER]: Refusing update from remote peer fd2576456e374554ad8d8c3b5db476e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:56.187747 23022 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:56.178810 22787 master_runner.cc:186] Error getting master registration for 127.0.97.61:36995: OK, instance_id { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" instance_seqno: 1790492395373009 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 06:59:56.196244 22787 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 06:59:56.198647 23047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } } peers { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } } peers { permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } } }
I20260927 06:59:56.199150 23047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:59:56.201596 22902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:56.203075 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:56.126553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47036 (local address 127.0.97.60:40333)
0927 06:59:56.126772 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:56.126777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:56.146407 (+ 19630us) server_negotiation.cc:407] Connection header received
0927 06:59:56.153044 (+ 6637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:56.153053 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:56.153151 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:56.153236 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:56.162144 (+ 8908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:56.163186 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:56.170209 (+ 7023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:56.170935 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:56.171033 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:56.181585 (+ 10552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:56.181595 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:56.181602 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:56.181652 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:56.186197 (+ 4545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:56.198391 (+ 12194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:56.198397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:56.198400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:56.198558 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:56.199441 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:56.199446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:56.199449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:56.199848 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:56.200107 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:56.202391 (+ 2284us) server_negotiation.cc:299] Negotiation successful
0927 06:59:56.202669 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 06:59:56.205101 22902 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d [term 1 FOLLOWER]: Refusing update from remote peer fd2576456e374554ad8d8c3b5db476e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:56.205574 22903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd2576456e374554ad8d8c3b5db476e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "36250d70cf8a438aa765885031a4da4d"
I20260927 06:59:56.206357 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:56.126855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.0.97.62:35007)
0927 06:59:56.127019 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:56.127025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:56.144620 (+ 17595us) server_negotiation.cc:407] Connection header received
0927 06:59:56.147606 (+ 2986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:56.147615 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:56.147729 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:56.147818 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:56.158785 (+ 10967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:56.159962 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:56.168256 (+ 8294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:56.169097 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:56.169205 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:56.187145 (+ 17940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:56.187155 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:56.187162 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:56.187218 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:56.191681 (+ 4463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:56.198993 (+ 7312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:56.198998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:56.199001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:56.199262 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:56.201517 (+ 2255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:56.201522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:56.201526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:56.201946 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:56.202208 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:56.205642 (+ 3434us) server_negotiation.cc:299] Negotiation successful
0927 06:59:56.205918 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 06:59:56.208724 22903 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd2576456e374554ad8d8c3b5db476e5 in term 1.
I20260927 06:59:56.209203 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:56.127562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60340 (local address 127.0.97.61:36995)
0927 06:59:56.127756 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:56.127762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:56.155826 (+ 28064us) server_negotiation.cc:407] Connection header received
0927 06:59:56.156067 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:56.156074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:56.156146 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:56.156209 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:56.165239 (+ 9030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:56.166374 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:56.172036 (+ 5662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:56.172838 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:56.172944 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:56.192838 (+ 19894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:56.192848 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:56.192856 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:56.192907 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:56.197194 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:56.200841 (+ 3647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:56.200846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:56.200849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:56.201023 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:56.204212 (+ 3189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:56.204217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:56.204221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:56.204557 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:56.204798 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:56.208598 (+ 3800us) server_negotiation.cc:299] Negotiation successful
0927 06:59:56.208884 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 06:59:56.210319 22928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:56.211241 23022 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:56.227665 23008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd2576456e374554ad8d8c3b5db476e5. Latest consensus state: current_term: 1 leader_uuid: "fd2576456e374554ad8d8c3b5db476e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } } peers { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } } peers { permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } } }
I20260927 06:59:56.228075 23008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:59:56.255378 23008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd2576456e374554ad8d8c3b5db476e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } } peers { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } } peers { permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } } }
I20260927 06:59:56.255705 23008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:59:56.258185 23041 catalog_manager.cc:1347] Generated new cluster ID: 0d13426971964eba8ff013992511b8b7
I20260927 06:59:56.258289 23041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:56.259414 23028 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd2576456e374554ad8d8c3b5db476e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } } peers { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } } peers { permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } } }
I20260927 06:59:56.259711 23028 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:56.263608 23047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd2576456e374554ad8d8c3b5db476e5. Latest consensus state: current_term: 1 leader_uuid: "fd2576456e374554ad8d8c3b5db476e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } } peers { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } } peers { permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } } }
I20260927 06:59:56.264010 23047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:59:56.280476 23060 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd2576456e374554ad8d8c3b5db476e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } } peers { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } } peers { permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } } }
I20260927 06:59:56.281899 23060 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:56.299583 23047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd2576456e374554ad8d8c3b5db476e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd2576456e374554ad8d8c3b5db476e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35007 } } peers { permanent_uuid: "36250d70cf8a438aa765885031a4da4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36995 } } peers { permanent_uuid: "24c91b0c14e04676b807aba7524f3813" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40333 } } }
I20260927 06:59:56.299913 23047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:59:56.308377 23041 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:56.310328 23041 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:56.321784 23068 catalog_manager.cc:1259] Loaded cluster ID: 0d13426971964eba8ff013992511b8b7
I20260927 06:59:56.321878 23068 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d: loading cluster ID for follower catalog manager: success
I20260927 06:59:56.325461 23068 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d: acquiring CA information for follower catalog manager: success
W20260927 06:59:56.326758 23068 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260927 06:59:56.337584 23041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd2576456e374554ad8d8c3b5db476e5: Generated new TSK 0
I20260927 06:59:56.338531 23041 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 06:59:56.456808 388 mini_kdc.cc:348] Time spent kdestroy: real 0.101s user 0.000s sys 0.021s
W20260927 06:59:56.518569 22932 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:56.497775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60350 (local address 127.0.97.61:36995)
0927 06:59:56.497953 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:56.497960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:56.515788 (+ 17828us) server_negotiation.cc:407] Connection header received
0927 06:59:56.516542 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:56.516553 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:56.516657 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:56.516739 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:56.518313 (+ 1574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60350: BlockingRecv error: recv got EOF from 127.0.0.1:60350 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 06:59:56.520138 22858 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:56.490300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.0.97.62:35007)
0927 06:59:56.490594 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:56.490601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:56.515333 (+ 24732us) server_negotiation.cc:407] Connection header received
0927 06:59:56.516953 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:56.516959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:56.517046 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:56.517129 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:56.519901 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58036: BlockingRecv error: recv got EOF from 127.0.0.1:58036 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 06:59:56.522696 23005 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:56.511377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47040 (local address 127.0.97.60:40333)
0927 06:59:56.511558 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:56.511564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:56.520806 (+ 9242us) server_negotiation.cc:407] Connection header received
0927 06:59:56.521372 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:56.521377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:56.521474 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:56.521550 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:56.522441 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47040: BlockingRecv error: recv got EOF from 127.0.0.1:47040 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 06:59:56.560860 22932 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:56.549407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60358 (local address 127.0.97.61:36995)
0927 06:59:56.549623 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:56.549629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:56.558579 (+ 8950us) server_negotiation.cc:407] Connection header received
0927 06:59:56.559148 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:56.559155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:56.559315 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:56.559410 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:56.560543 (+ 1133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60358: BlockingRecv error: recv got EOF from 127.0.0.1:60358 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260927 06:59:56.609491 22932 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:56.596740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.0.97.61:36995)
0927 06:59:56.596906 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:56.596912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:56.607564 (+ 10652us) server_negotiation.cc:407] Connection header received
0927 06:59:56.608095 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:56.608102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:56.608211 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:56.608285 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:56.609253 (+ 968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60366: BlockingRecv error: recv got EOF from 127.0.0.1:60366 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 06:59:56.659030 22932 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:56.646128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60382 (local address 127.0.97.61:36995)
0927 06:59:56.646340 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:56.646347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:56.657119 (+ 10772us) server_negotiation.cc:407] Connection header received
0927 06:59:56.657599 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:56.657605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:56.657717 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:56.657794 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:56.658765 (+ 971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60382: BlockingRecv error: recv got EOF from 127.0.0.1:60382 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 06:59:56.669413 22932 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:56.667320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.0.97.61:36995)
0927 06:59:56.667603 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:56.667612 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:56.668065 (+ 453us) server_negotiation.cc:407] Connection header received
0927 06:59:56.668267 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:56.668279 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:56.668351 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:56.668456 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:56.669158 (+ 702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60392: BlockingRecv error: recv got EOF from 127.0.0.1:60392 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
W20260927 06:59:56.682389 22932 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:56.680356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.0.97.61:36995)
0927 06:59:56.680579 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:56.680588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:56.681064 (+ 476us) server_negotiation.cc:407] Connection header received
0927 06:59:56.681267 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:56.681271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:56.681337 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:56.681441 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:56.682150 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60406: BlockingRecv error: recv got EOF from 127.0.0.1:60406 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 06:59:56.703442 22932 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:56.701348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60408 (local address 127.0.97.61:36995)
0927 06:59:56.701590 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:56.701602 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:56.702102 (+ 500us) server_negotiation.cc:407] Connection header received
0927 06:59:56.702355 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:56.702360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:56.702423 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:56.702530 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:56.703199 (+ 669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60408: BlockingRecv error: recv got EOF from 127.0.0.1:60408 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 06:59:56.740888 22932 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:56.738640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60420 (local address 127.0.97.61:36995)
0927 06:59:56.738862 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:56.738871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:56.739446 (+ 575us) server_negotiation.cc:407] Connection header received
0927 06:59:56.739683 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:56.739688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:56.739772 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:56.739889 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:56.740589 (+ 700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60420: BlockingRecv error: recv got EOF from 127.0.0.1:60420 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 06:59:56.814258 22932 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:56.812159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.0.97.61:36995)
0927 06:59:56.812397 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:56.812406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:56.812776 (+ 370us) server_negotiation.cc:407] Connection header received
0927 06:59:56.813005 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:56.813009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:56.813075 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:56.813203 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:56.814039 (+ 836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60424: BlockingRecv error: recv got EOF from 127.0.0.1:60424 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 06:59:56.947949 22932 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:56.945915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60436 (local address 127.0.97.61:36995)
0927 06:59:56.946171 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:56.946179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:56.946527 (+ 348us) server_negotiation.cc:407] Connection header received
0927 06:59:56.946774 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:56.946778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:56.946845 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:56.946957 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:56.947705 (+ 748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60436: BlockingRecv error: recv got EOF from 127.0.0.1:60436 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 06:59:56.981881 23019 catalog_manager.cc:1259] Loaded cluster ID: 0d13426971964eba8ff013992511b8b7
I20260927 06:59:56.981968 23019 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813: loading cluster ID for follower catalog manager: success
I20260927 06:59:56.984477 23019 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813: acquiring CA information for follower catalog manager: success
I20260927 06:59:56.986114 23019 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 24c91b0c14e04676b807aba7524f3813: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 06:59:57.213616 22932 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:57.211807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60444 (local address 127.0.97.61:36995)
0927 06:59:57.211995 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:57.212003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:57.212347 (+ 344us) server_negotiation.cc:407] Connection header received
0927 06:59:57.212573 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:57.212578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:57.212648 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:57.212741 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:57.213432 (+ 691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60444: BlockingRecv error: recv got EOF from 127.0.0.1:60444 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 06:59:57.329265 23068 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 36250d70cf8a438aa765885031a4da4d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 06:59:57.737408 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:57.731586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60454 (local address 127.0.97.61:36995)
0927 06:59:57.732112 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:57.732118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:57.732251 (+ 133us) server_negotiation.cc:407] Connection header received
0927 06:59:57.732494 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:57.732498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:57.732652 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:57.732811 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:57.733288 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:57.733962 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:57.735302 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:57.735713 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:57.735844 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:57.736306 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:57.736783 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:57.736934 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:57.737029 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 06:59:57.737104 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":276,"threads_started":1}
I20260927 06:59:57.748086 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:57.743073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47056 (local address 127.0.97.60:40333)
0927 06:59:57.743594 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:57.743598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:57.743614 (+ 16us) server_negotiation.cc:407] Connection header received
0927 06:59:57.743802 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:57.743810 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:57.743969 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:57.744097 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:57.744426 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:57.745077 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:57.746294 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:57.746683 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:57.746771 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:57.747124 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:57.747575 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:57.747671 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:57.747733 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 06:59:57.747788 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":272,"threads_started":1}
I20260927 06:59:57.771291 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:57.754280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.0.97.61:36995)
0927 06:59:57.754623 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:57.754630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:57.765139 (+ 10509us) server_negotiation.cc:407] Connection header received
0927 06:59:57.765469 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:57.765477 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:57.765583 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:57.765655 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:57.766335 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:57.767298 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:57.769268 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:57.769735 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:57.769867 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:57.770495 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:57.770623 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:57.770703 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:57.770892 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 06:59:57.770983 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
I20260927 06:59:57.771322 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:57.754273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47068 (local address 127.0.97.60:40333)
0927 06:59:57.754461 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:57.754470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:57.765655 (+ 11185us) server_negotiation.cc:407] Connection header received
0927 06:59:57.765825 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:57.765831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:57.765919 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:57.766033 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:57.766751 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:57.767739 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:57.769395 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:57.770021 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:57.770140 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:57.770582 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:57.770699 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:57.770786 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:57.770976 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 06:59:57.771050 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 06:59:57.773108 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:57.753829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58040 (local address 127.0.97.62:35007)
0927 06:59:57.766158 (+ 12329us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:57.766164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:57.766180 (+ 16us) server_negotiation.cc:407] Connection header received
0927 06:59:57.766300 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:57.766306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:57.766517 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:57.766695 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:57.767209 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:57.768199 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:57.770508 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:57.771189 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:57.771779 (+ 590us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:57.771984 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:57.772420 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:57.772556 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:57.772716 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 06:59:57.772795 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12208,"thread_start_us":12082,"threads_started":1}
I20260927 06:59:57.777447 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 22787
I20260927 06:59:57.778458 22849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:57.886098 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 22787
I20260927 06:59:57.906456 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 22861
I20260927 06:59:57.907510 22923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:58.009783 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 22861
I20260927 06:59:58.029655 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 22935
I20260927 06:59:58.030686 22997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 06:59:58.133461 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 22935
2026-09-27T06:59:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4946 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:58 dist-test-slave-f892 krb5kdc[23154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:58 dist-test-slave-f892 krb5kdc[23154](info): set up 2 sockets
Sep 27 06:59:58 dist-test-slave-f892 krb5kdc[23154](info): commencing operation
krb5kdc: starting...
W20260927 06:59:59.222450 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/test-user.keytab.
Sep 27 06:59:59 dist-test-slave-f892 krb5kdc[23154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 06:59:59.456465 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35653
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35653,127.0.97.61:41081,127.0.97.60:43633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:59.765375 23167 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 06:59:59.765677 23167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:59.765739 23167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:59.765810 23167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:59.772907 23167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:59.772987 23167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:59.773042 23167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:59.773092 23167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:59.773145 23167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:59.782507 23167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35653,127.0.97.61:41081,127.0.97.60:43633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 06:59:59.783604 23167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:59.784798 23167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:59.795655 23172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:59.795708 23175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:59.795746 23173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:59.796600 23167 server_base.cc:1034] running on GCE node
I20260927 06:59:59.797267 23167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:59.798542 23167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:59.799768 23167 hybrid_clock.cc:648] HybridClock initialized: now 1790492399799698 us; error 50 us; skew 500 ppm
I20260927 06:59:59.800076 23167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 06:59:59 dist-test-slave-f892 krb5kdc[23154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:59.802935 23167 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 06:59:59.804229 23167 webserver.cc:460] Webserver started at http://127.0.97.62:37185/ using document root <none> and password file <none>
I20260927 06:59:59.804860 23167 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:59.805006 23167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:59.805297 23167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:59.808110 23167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-0/data/instance:
uuid: "3829c3c7c997488db296e6ac759f7949"
format_stamp: "Formatted at 2026-09-27 06:59:59 on dist-test-slave-f892"
server_key: "bca2c08783869d69afc0c849d779e0d6"
server_key_iv: "aba0b6ed0b6563bc05d96982191425f3"
server_key_version: "encryptionkey@0"
I20260927 06:59:59.808851 23167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-0/wal/instance:
uuid: "3829c3c7c997488db296e6ac759f7949"
format_stamp: "Formatted at 2026-09-27 06:59:59 on dist-test-slave-f892"
server_key: "bca2c08783869d69afc0c849d779e0d6"
server_key_iv: "aba0b6ed0b6563bc05d96982191425f3"
server_key_version: "encryptionkey@0"
I20260927 06:59:59.813529 23167 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 06:59:59.816656 23182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:59.817795 23167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 06:59:59.817961 23167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-0/wal
uuid: "3829c3c7c997488db296e6ac759f7949"
format_stamp: "Formatted at 2026-09-27 06:59:59 on dist-test-slave-f892"
server_key: "bca2c08783869d69afc0c849d779e0d6"
server_key_iv: "aba0b6ed0b6563bc05d96982191425f3"
server_key_version: "encryptionkey@0"
I20260927 06:59:59.818213 23167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:59.847499 23167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:59.851142 23167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:59.851416 23167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:59.868746 23167 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35653
I20260927 06:59:59.868769 23233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35653 every 8 connection(s)
I20260927 06:59:59.870041 23167 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:59.873448 23234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:59.878641 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 23167
I20260927 06:59:59.878808 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-0/wal/instance
I20260927 06:59:59.879292 388 external_mini_cluster.cc:1428] Setting key 9688eaada9acb74385eae263fd53cafc
I20260927 06:59:59.879767 23234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 06:59:59 dist-test-slave-f892 krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 06:59:59.903012 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:59.883056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35653 (local address 127.0.0.1:56856)
0927 06:59:59.884219 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:59.884270 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:59.886949 (+ 2679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:59.887758 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:59.887770 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:59.888549 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:59.888945 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:59.888962 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:59.890540 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:59.890544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:59.891634 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:59.891645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:59.892564 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:59.892570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:59.892779 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:59.893453 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:59.893474 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:59.895774 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:59.898885 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:59.898897 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:59.898908 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:59.899256 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:59.899792 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:59.899795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:59.899797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:59.899929 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:59.900657 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:59.900664 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:59.900873 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 06:59:59.901111 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 06:59:59.901650 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":129,"spinlock_wait_cycles":11136,"thread_start_us":719,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260927 06:59:59.903043 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:59.883057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56856 (local address 127.0.97.62:35653)
0927 06:59:59.883934 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:59.883956 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:59.884392 (+ 436us) server_negotiation.cc:407] Connection header received
0927 06:59:59.887277 (+ 2885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:59.887283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:59.887496 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:59.887669 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:59.889152 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:59.890387 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:59.891803 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:59.892335 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:59.892522 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:59.896013 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:59.896021 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:59.896031 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:59.896057 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:59.898758 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:59.899396 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:59.899405 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:59.899414 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:59.899540 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:59.900030 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:59.900037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:59.900043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:59.900514 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:59.900715 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:59.901236 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 06:59:59.901534 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":623,"thread_start_us":404,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260927 06:59:59.906265 23234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } has no permanent_uuid. Determining permanent_uuid...
I20260927 06:59:59.907377 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:59.906987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41081 (local address 127.0.0.1:47040)
0927 06:59:59.907141 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:59.907253 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 06:59:59.907598 23183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41081: connect: Connection refused (error 111)
I20260927 06:59:59.909310 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:59.908840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41081 (local address 127.0.0.1:47050)
0927 06:59:59.909059 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:59.909149 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260927 06:59:59.909505 23234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41081: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41081: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 06:59:59.937062 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41081
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35653,127.0.97.61:41081,127.0.97.60:43633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:59:59.958854 23234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } attempt: 1
I20260927 06:59:59.960038 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:59.959548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41081 (local address 127.0.0.1:47052)
0927 06:59:59.959759 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:59.959883 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 06:59:59.960978 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:59.960614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41081 (local address 127.0.0.1:47066)
0927 06:59:59.960754 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:59.960837 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 06:59:59.961172 23234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41081: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41081: connect: Connection refused (error 111)
I20260927 07:00:00.029443 23234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } attempt: 2
I20260927 07:00:00.030897 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.030242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41081 (local address 127.0.0.1:47068)
0927 07:00:00.030567 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.030705 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 07:00:00.031960 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.031528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41081 (local address 127.0.0.1:47070)
0927 07:00:00.031727 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.031807 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260927 07:00:00.032164 23234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41081: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41081: connect: Connection refused (error 111)
I20260927 07:00:00.123425 23234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } attempt: 3
I20260927 07:00:00.124650 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.124095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41081 (local address 127.0.0.1:47084)
0927 07:00:00.124339 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.124466 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 07:00:00.125654 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.125254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41081 (local address 127.0.0.1:47094)
0927 07:00:00.125407 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.125487 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 07:00:00.125867 23234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41081: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41081: connect: Connection refused (error 111)
W20260927 07:00:00.256567 23241 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:00:00.256868 23241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:00.256940 23241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:00.257020 23241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:00.264092 23241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:00:00.264173 23241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:00.264227 23241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:00.264277 23241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:00:00.264335 23241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:00:00.269100 23234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } attempt: 4
I20260927 07:00:00.270346 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.269727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41081 (local address 127.0.0.1:47098)
0927 07:00:00.270083 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.270185 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260927 07:00:00.271317 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.270937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41081 (local address 127.0.0.1:47112)
0927 07:00:00.271087 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.271177 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 07:00:00.271533 23234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41081: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41081: connect: Connection refused (error 111)
I20260927 07:00:00.274886 23241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35653,127.0.97.61:41081,127.0.97.60:43633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:00:00.276137 23241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:00.277418 23241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:00.288842 23247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:00.288879 23249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:00.289160 23246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:00.290167 23241 server_base.cc:1034] running on GCE node
I20260927 07:00:00.290956 23241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:00.292280 23241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:00.293505 23241 hybrid_clock.cc:648] HybridClock initialized: now 1790492400293441 us; error 35 us; skew 500 ppm
I20260927 07:00:00.293850 23241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:00:00 dist-test-slave-f892 krb5kdc[23154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:00:00.296973 23241 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 07:00:00.299079 23241 webserver.cc:460] Webserver started at http://127.0.97.61:44895/ using document root <none> and password file <none>
I20260927 07:00:00.299769 23241 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:00.299901 23241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:00.300221 23241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:00.303129 23241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-1/data/instance:
uuid: "f438704b99b8401c9dd69b02a9bd33dc"
format_stamp: "Formatted at 2026-09-27 07:00:00 on dist-test-slave-f892"
server_key: "fed9623f0fdd8a74ff2a8e67b7dbbda4"
server_key_iv: "7d998d179f0fc40532d9bc8349d63ab6"
server_key_version: "encryptionkey@0"
I20260927 07:00:00.303894 23241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-1/wal/instance:
uuid: "f438704b99b8401c9dd69b02a9bd33dc"
format_stamp: "Formatted at 2026-09-27 07:00:00 on dist-test-slave-f892"
server_key: "fed9623f0fdd8a74ff2a8e67b7dbbda4"
server_key_iv: "7d998d179f0fc40532d9bc8349d63ab6"
server_key_version: "encryptionkey@0"
I20260927 07:00:00.308899 23241 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 07:00:00.312295 23256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:00.313506 23241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 07:00:00.313696 23241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-1/wal
uuid: "f438704b99b8401c9dd69b02a9bd33dc"
format_stamp: "Formatted at 2026-09-27 07:00:00 on dist-test-slave-f892"
server_key: "fed9623f0fdd8a74ff2a8e67b7dbbda4"
server_key_iv: "7d998d179f0fc40532d9bc8349d63ab6"
server_key_version: "encryptionkey@0"
I20260927 07:00:00.313927 23241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:00.325917 23241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:00.330113 23241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:00.330370 23241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:00.348052 23241 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:41081
I20260927 07:00:00.348069 23307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:41081 every 8 connection(s)
I20260927 07:00:00.349352 23241 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-1/data/info.pb
I20260927 07:00:00.349763 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 23241
I20260927 07:00:00.349934 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-1/wal/instance
I20260927 07:00:00.350585 388 external_mini_cluster.cc:1428] Setting key d4f3481525f7a05ed500a44d9df1978e
I20260927 07:00:00.352928 23308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:00.359163 23308 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Sep 27 07:00:00 dist-test-slave-f892 krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260927 07:00:00.379496 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.361689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.0.97.62:35653)
0927 07:00:00.361909 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:00.361915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:00.363105 (+ 1190us) server_negotiation.cc:407] Connection header received
0927 07:00:00.364904 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:00.364912 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:00.365007 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:00.365140 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:00.366604 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.367799 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.369234 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.370101 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.370252 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:00.373812 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:00.373817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:00.373820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:00.373853 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:00.376586 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:00.377297 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:00.377305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:00.377310 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:00.377426 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:00.377813 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:00.377818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:00.377821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:00.378169 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:00.378326 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:00.378952 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 07:00:00.379167 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 07:00:00.380365 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.362095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35653 (local address 127.0.0.1:56872)
0927 07:00:00.362926 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.362977 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:00.364645 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:00.365318 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:00.365325 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:00.365828 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:00.366423 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:00.366441 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.367944 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.367948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:00.369080 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:00.369090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.370301 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.370306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:00.370504 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:00.371178 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:00.371203 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:00.373594 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:00.376740 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:00.376748 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:00.376762 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:00.377168 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:00.377532 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:00.377539 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:00.377544 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:00.377706 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:00.378286 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:00.378293 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:00.378605 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 07:00:00.378875 (+ 270us) client_negotiation.cc:240] Negotiation successful
0927 07:00:00.379352 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":365,"threads_started":1}
I20260927 07:00:00.382054 23308 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:00:00 dist-test-slave-f892 krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 07:00:00.398380 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.382849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41081 (local address 127.0.0.1:47122)
0927 07:00:00.383017 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.383038 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:00.383126 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:00.384372 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:00.384375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:00.384696 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:00.384828 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:00.384835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.386869 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.386872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:00.387681 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:00.387691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.388858 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.388862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:00.389000 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:00.389571 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:00.389587 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:00.391859 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:00.396104 (+ 4245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:00.396108 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:00.396110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:00.396438 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:00.396836 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:00.396839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:00.396841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:00.396960 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:00.397647 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:00.397649 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:00.397765 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 07:00:00.397865 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 07:00:00.398097 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":3}
I20260927 07:00:00.398770 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.383124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47122 (local address 127.0.97.61:41081)
0927 07:00:00.383867 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:00.383895 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:00.383927 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:00:00.383990 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:00.384001 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:00.384215 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:00.384359 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:00.384995 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.386731 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.387819 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.388730 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.388863 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:00.392076 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:00.392091 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:00.392110 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:00.392155 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:00.395979 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:00.396565 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:00.396575 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:00.396580 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:00.396726 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:00.397079 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:00.397086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:00.397091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:00.397530 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:00.397757 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:00.398083 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 07:00:00.398366 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":384,"threads_started":1}
I20260927 07:00:00.401093 23308 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:00:00.402150 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.401658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:39970)
0927 07:00:00.401814 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.401923 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 07:00:00.402375 23260 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
I20260927 07:00:00.403842 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.403550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:39982)
0927 07:00:00.403685 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.403752 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 07:00:00.404052 23308 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43633: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 07:00:00.408525 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
/tmp/dist-test-tasktaM5HI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:43633
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35653,127.0.97.61:41081,127.0.97.60:43633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:00:00.453330 23308 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } attempt: 1
I20260927 07:00:00.454449 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.453970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:39990)
0927 07:00:00.454204 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.454299 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260927 07:00:00.455389 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.455057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:39998)
0927 07:00:00.455218 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.455293 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 07:00:00.455610 23308 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43633: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
I20260927 07:00:00.523798 23308 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } attempt: 2
I20260927 07:00:00.524987 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.524432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:40004)
0927 07:00:00.524660 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.524773 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 07:00:00.526090 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.525677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:40010)
0927 07:00:00.525826 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.525918 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 07:00:00.526319 23308 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43633: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
I20260927 07:00:00.570796 23234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } attempt: 5
Sep 27 07:00:00 dist-test-slave-f892 krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 07:00:00.585234 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.571741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47126 (local address 127.0.97.61:41081)
0927 07:00:00.571900 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:00.571905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:00.571924 (+ 19us) server_negotiation.cc:407] Connection header received
0927 07:00:00.572066 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:00.572070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:00.572146 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:00.572223 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:00.573202 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.573992 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.575719 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.576156 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.576244 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:00.579778 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:00.579783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:00.579786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:00.579813 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:00.582495 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:00.583399 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:00.583403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:00.583405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:00.583522 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:00.583976 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:00.583979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:00.583981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:00.584242 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:00.584346 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:00.584843 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 07:00:00.585000 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 07:00:00.585552 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.571464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41081 (local address 127.0.0.1:47126)
0927 07:00:00.571742 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.571767 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:00.571921 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:00.572299 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:00.572304 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:00.572811 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:00.573052 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:00.573066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.574197 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.574203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:00.575582 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:00.575599 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.576313 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.576319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:00.576495 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:00.577106 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:00.577128 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:00.579600 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:00.582642 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:00.582649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:00.582656 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:00.583283 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:00.583640 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:00.583648 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:00.583652 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:00.583845 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:00.584375 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:00.584381 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:00.584583 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 07:00:00.584774 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 07:00:00.585141 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 07:00:00.586664 23234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:00:00.587666 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.587231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:40012)
0927 07:00:00.587403 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.587494 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 07:00:00.588647 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.588296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:40020)
0927 07:00:00.588453 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.588532 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 07:00:00.588843 23234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43633: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
I20260927 07:00:00.617586 23308 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } attempt: 3
I20260927 07:00:00.619014 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.618279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:40032)
0927 07:00:00.618585 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.618733 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 07:00:00.620353 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.619777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:40044)
0927 07:00:00.620065 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.620176 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260927 07:00:00.620653 23308 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43633: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
I20260927 07:00:00.640079 23234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } attempt: 1
I20260927 07:00:00.641294 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.640738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:40054)
0927 07:00:00.640974 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.641086 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 07:00:00.642350 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.641920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:40056)
0927 07:00:00.642081 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.642170 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 07:00:00.642550 23234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43633: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
I20260927 07:00:00.710781 23234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } attempt: 2
I20260927 07:00:00.712040 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.711464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:40060)
0927 07:00:00.711715 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.711836 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 07:00:00.713106 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.712692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:40066)
0927 07:00:00.712865 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.712956 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 07:00:00.713323 23234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43633: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
W20260927 07:00:00.741545 23315 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:00:00.741849 23315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:00.741921 23315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:00.742017 23315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:00.749177 23315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:00:00.749262 23315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:00.749317 23315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:00.749365 23315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:00:00.749418 23315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:00:00.759060 23315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35653,127.0.97.61:41081,127.0.97.60:43633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:43633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:00:00.760226 23315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:00.761458 23315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:00:00.763865 23308 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } attempt: 4
I20260927 07:00:00.765046 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.764458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:40080)
0927 07:00:00.764712 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.764819 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 07:00:00.766103 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.765696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:40086)
0927 07:00:00.765864 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.765947 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 07:00:00.766371 23308 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43633: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
W20260927 07:00:00.773473 23323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:00.773551 23320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:00.773512 23321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:00.774350 23315 server_base.cc:1034] running on GCE node
I20260927 07:00:00.775049 23315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:00.776315 23315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:00.777568 23315 hybrid_clock.cc:648] HybridClock initialized: now 1790492400777499 us; error 57 us; skew 500 ppm
I20260927 07:00:00.777884 23315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:00:00 dist-test-slave-f892 krb5kdc[23154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:00:00.781165 23315 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 07:00:00.782630 23315 webserver.cc:460] Webserver started at http://127.0.97.60:46631/ using document root <none> and password file <none>
I20260927 07:00:00.783241 23315 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:00.783324 23315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:00.783574 23315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:00.786375 23315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-2/data/instance:
uuid: "d962f584390441b2b7cc1d622a629c36"
format_stamp: "Formatted at 2026-09-27 07:00:00 on dist-test-slave-f892"
server_key: "414a8923ec1e647409870663e9c245e3"
server_key_iv: "bbc8c3f981a056a07f355ebac3d802bc"
server_key_version: "encryptionkey@0"
I20260927 07:00:00.787075 23315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-2/wal/instance:
uuid: "d962f584390441b2b7cc1d622a629c36"
format_stamp: "Formatted at 2026-09-27 07:00:00 on dist-test-slave-f892"
server_key: "414a8923ec1e647409870663e9c245e3"
server_key_iv: "bbc8c3f981a056a07f355ebac3d802bc"
server_key_version: "encryptionkey@0"
I20260927 07:00:00.791780 23315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 07:00:00.794981 23330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:00.796202 23315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 07:00:00.796335 23315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-2/wal
uuid: "d962f584390441b2b7cc1d622a629c36"
format_stamp: "Formatted at 2026-09-27 07:00:00 on dist-test-slave-f892"
server_key: "414a8923ec1e647409870663e9c245e3"
server_key_iv: "bbc8c3f981a056a07f355ebac3d802bc"
server_key_version: "encryptionkey@0"
I20260927 07:00:00.796496 23315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:00.812216 23315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:00.815863 23315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:00.816078 23315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:00.819516 23234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } attempt: 3
I20260927 07:00:00.820663 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.820111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:40092)
0927 07:00:00.820385 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.820490 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 07:00:00.821780 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.821392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:40106)
0927 07:00:00.821608 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.821691 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260927 07:00:00.822104 23234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43633: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43633: connect: Connection refused (error 111)
I20260927 07:00:00.834514 23315 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:43633
I20260927 07:00:00.834589 23381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:43633 every 8 connection(s)
I20260927 07:00:00.835774 23315 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-2/data/info.pb
I20260927 07:00:00.839237 23382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:00.842633 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu as pid 23315
I20260927 07:00:00.842813 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktaM5HI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492047466241-388-0/minicluster-data/master-2/wal/instance
I20260927 07:00:00.843348 388 external_mini_cluster.cc:1428] Setting key 6b60a309c6344e5e23ad2c49c3e86fc9
I20260927 07:00:00.846917 23382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:00:00 dist-test-slave-f892 krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 07:00:00 dist-test-slave-f892 krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 07:00:00.864049 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.849795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56882 (local address 127.0.97.62:35653)
0927 07:00:00.850045 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:00.850052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:00.850804 (+ 752us) server_negotiation.cc:407] Connection header received
0927 07:00:00.850975 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:00.850980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:00.851057 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:00.851153 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:00.852201 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.852869 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.854526 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.854947 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.855095 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:00.858432 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:00.858438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:00.858441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:00.858472 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:00.861273 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:00.862087 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:00.862091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:00.862093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:00.862217 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:00.862631 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:00.862635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:00.862638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:00.862969 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:00.863145 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:00.863624 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 07:00:00.863780 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 07:00:00.864634 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.845148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.0.97.60:43633)
0927 07:00:00.845863 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:00.845875 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:00.845913 (+ 38us) server_negotiation.cc:407] Connection header received
0927 07:00:00.847556 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:00.847575 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:00.847958 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:00.848331 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:00.849708 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.851360 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.852636 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.853288 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.853476 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:00.857303 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:00.857310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:00.857320 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:00.857347 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:00.860328 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:00.861095 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:00.861100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:00.861106 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:00.861244 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:00.861630 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:00.861634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:00.861638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:00.862248 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:00.862578 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:00.862881 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 07:00:00.863290 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":330,"threads_started":1}
I20260927 07:00:00.864653 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.849852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35653 (local address 127.0.0.1:56882)
0927 07:00:00.850656 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.850702 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:00.850848 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:00.851203 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:00.851208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:00.851819 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:00.852052 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:00.852060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.853037 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.853047 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:00.854366 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:00.854378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.855175 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.855184 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:00.855353 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:00.855943 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:00.855965 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:00.858233 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:00.861414 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:00.861421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:00.861426 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:00.861920 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:00.862325 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:00.862331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:00.862335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:00.862508 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:00.863142 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:00.863149 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:00.863335 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 07:00:00.863518 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 07:00:00.863794 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":415,"threads_started":1}
I20260927 07:00:00.866288 23382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:00:00 dist-test-slave-f892 krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 27 07:00:00 dist-test-slave-f892 krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 07:00:00.882200 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.867109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41081 (local address 127.0.0.1:47136)
0927 07:00:00.867283 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.867301 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:00.867403 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:00.867855 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:00.867858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:00.868313 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:00.868496 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:00.868507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.869878 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.869883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:00.870817 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:00.870828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.871819 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.871825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:00.872037 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:00.872651 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:00.872668 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:00.875203 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:00.879289 (+ 4086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:00.879294 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:00.879297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:00.879786 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:00.880307 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:00.880310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:00.880313 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:00.880447 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:00.881154 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:00.881161 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:00.881321 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 07:00:00.881483 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 07:00:00.881776 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 07:00:00.883106 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.867145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.0.97.61:41081)
0927 07:00:00.867329 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:00.867334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:00.867372 (+ 38us) server_negotiation.cc:407] Connection header received
0927 07:00:00.867519 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:00.867525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:00.867603 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:00.867697 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:00.868657 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.869742 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.871051 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.871670 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.871786 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:00.875400 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:00.875407 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:00.875412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:00.875449 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:00.878859 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:00.879933 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:00.879964 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:00.879971 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:00.880160 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:00.880559 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:00.880566 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:00.880569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:00.880930 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:00.881091 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:00.882510 (+ 1419us) server_negotiation.cc:299] Negotiation successful
0927 07:00:00.882733 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 07:00:00.884232 23382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:00:00.885653 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.867703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56890 (local address 127.0.97.62:35653)
0927 07:00:00.867860 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:00.867866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:00.867883 (+ 17us) server_negotiation.cc:407] Connection header received
0927 07:00:00.868061 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:00.868066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:00.868136 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:00.868329 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:00.869679 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.870789 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.872155 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.872793 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.872925 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:00.878129 (+ 5204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:00.878139 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:00.878145 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:00.878187 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:00.882565 (+ 4378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:00.883372 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:00.883376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:00.883378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:00.883494 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:00.883969 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:00.883972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:00.883974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:00.884330 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:00.884596 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:00.885116 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 07:00:00.885339 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 07:00:00 dist-test-slave-f892 krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 07:00:00 dist-test-slave-f892 krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 07:00:00.896957 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.885160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.0.97.60:43633)
0927 07:00:00.885305 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:00.885312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:00.885331 (+ 19us) server_negotiation.cc:407] Connection header received
0927 07:00:00.885362 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:00.885364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:00.885409 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:00.885512 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:00.886497 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.887465 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.888571 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.888961 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.889095 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:00.891967 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:00.891972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:00.891975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:00.892000 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:00.894612 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:00.895243 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:00.895248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:00.895253 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:00.895385 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:00.895744 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:00.895747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:00.895749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:00.896001 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:00.896106 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:00.896558 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 07:00:00.896728 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":49}
I20260927 07:00:00.897460 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.884917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:40124)
0927 07:00:00.885095 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.885113 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:00.885205 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:00.885673 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:00.885676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:00.886208 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:00.886363 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:00.886372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.887584 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.887592 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:00.888427 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:00.888437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.889100 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.889105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:00.889344 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:00.889906 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:00.889921 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:00.891801 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:00.894739 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:00.894742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:00.894745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:00.895127 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:00.895499 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:00.895502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:00.895504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:00.895618 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:00.896118 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:00.896125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:00.896284 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 07:00:00.896454 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 07:00:00.896916 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":116}
I20260927 07:00:00.901786 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.887328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47140 (local address 127.0.97.61:41081)
0927 07:00:00.887502 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:00.887507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:00.887523 (+ 16us) server_negotiation.cc:407] Connection header received
0927 07:00:00.887657 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:00.887662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:00.887730 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:00.887827 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:00.888942 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.889617 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.890849 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:00.891274 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:00.891401 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:00.895753 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:00.895759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:00.895763 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:00.895801 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:00.899272 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:00.900013 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:00.900017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:00.900019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:00.900150 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:00.900533 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:00.900536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:00.900538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:00.900814 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:00.901063 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:00.901357 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 07:00:00.901521 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 07:00:00.903313 23382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36: Bootstrap starting.
I20260927 07:00:00.906399 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:00:00.906963 23382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:00.907951 23382 log.cc:824] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36: Log is configured to *not* fsync() on all Append() calls
I20260927 07:00:00.911013 23382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36: No bootstrap required, opened a new log
I20260927 07:00:00.908785 23209 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3829c3c7c997488db296e6ac759f7949"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56890
I20260927 07:00:00.920279 23382 raft_consensus.cc:348] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } } peers { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } } peers { permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } }
I20260927 07:00:00.920580 23382 raft_consensus.cc:729] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d962f584390441b2b7cc1d622a629c36, State: Initialized, Role: FOLLOWER
I20260927 07:00:00.921125 23382 consensus_queue.cc:260] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } } peers { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } } peers { permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } }
I20260927 07:00:00.922129 23388 sys_catalog.cc:455] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } } peers { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } } peers { permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } } }
I20260927 07:00:00.922598 23388 sys_catalog.cc:458] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:00:00.923251 23382 sys_catalog.cc:564] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 07:00:00.932601 23399 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:00:00.932741 23399 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:00:00.930621 23315 master_runner.cc:186] Error getting master registration for 127.0.97.62:35653: OK, instance_id { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" instance_seqno: 1790492399863794 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:00:00.938571 23315 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 07:00:00.999360 23234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } attempt: 4
Sep 27 07:00:01 dist-test-slave-f892 krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 07:00:01.016189 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:01.000158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:40128)
0927 07:00:01.000342 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:01.000363 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:01.000480 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:01.000861 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:01.000866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:01.001362 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:01.001573 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:01.001585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:01.003086 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:01.003091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:01.004621 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:01.004640 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:01.005641 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:01.005648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:01.005844 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:01.006601 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:01.006627 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:01.009324 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:01.013497 (+ 4173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:01.013501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:01.013503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:01.013877 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:01.014281 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:01.014292 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:01.014295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:01.014514 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:01.015205 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:01.015210 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:01.015365 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 07:00:01.015503 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 07:00:01.015786 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 07:00:01.016217 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:01.000286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40128 (local address 127.0.97.60:43633)
0927 07:00:01.000490 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:01.000496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:01.000518 (+ 22us) server_negotiation.cc:407] Connection header received
0927 07:00:01.000628 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:01.000637 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:01.000713 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:01.000829 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:01.001732 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:01.002936 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:01.004783 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:01.005489 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:01.005615 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:01.009526 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:01.009537 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:01.009542 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:01.009581 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:01.013332 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:01.013990 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:01.013996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:01.014000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:01.014162 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:01.014627 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:01.014632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:01.014636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:01.015062 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:01.015224 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:01.015627 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 07:00:01.015851 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 07:00:01.022250 23234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949: Bootstrap starting.
I20260927 07:00:01.026340 23234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:01.027618 23234 log.cc:824] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949: Log is configured to *not* fsync() on all Append() calls
I20260927 07:00:01.030949 23234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949: No bootstrap required, opened a new log
I20260927 07:00:01.038512 23234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } } peers { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } } peers { permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } }
I20260927 07:00:01.038923 23234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3829c3c7c997488db296e6ac759f7949, State: Initialized, Role: FOLLOWER
I20260927 07:00:01.039419 23234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } } peers { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } } peers { permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } }
I20260927 07:00:01.040422 23402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } } peers { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } } peers { permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } } }
I20260927 07:00:01.040781 23402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:00:01.041622 23234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:00:01.041675 23209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 07:00:01.041894 23209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:01.044754 23209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } } peers { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } } peers { permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } }
I20260927 07:00:01.046962 23357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3829c3c7c997488db296e6ac759f7949" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d962f584390441b2b7cc1d622a629c36"
I20260927 07:00:01.047428 23357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:01.050978 23357 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3829c3c7c997488db296e6ac759f7949 in term 1.
I20260927 07:00:01.051584 23186 leader_election.cc:302] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3829c3c7c997488db296e6ac759f7949, d962f584390441b2b7cc1d622a629c36; no voters:
I20260927 07:00:01.051998 23209 leader_election.cc:288] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [CANDIDATE]: Term 1 election: Requested vote from peers f438704b99b8401c9dd69b02a9bd33dc (127.0.97.61:41081), d962f584390441b2b7cc1d622a629c36 (127.0.97.60:43633)
I20260927 07:00:01.052028 23402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:00:01.063485 23402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [term 1 LEADER]: Becoming Leader. State: Replica: 3829c3c7c997488db296e6ac759f7949, State: Running, Role: LEADER
I20260927 07:00:01.064673 23402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } } peers { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } } peers { permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } }
I20260927 07:00:01.065598 23308 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } attempt: 5
W20260927 07:00:01.068444 23336 tablet.cc:2369] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 07:00:01 dist-test-slave-f892 krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 07:00:01.092257 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:01.066450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43633 (local address 127.0.0.1:40144)
0927 07:00:01.066918 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:01.066944 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:01.067097 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:01.069488 (+ 2391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:01.069497 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:01.070201 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:01.070502 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:01.070519 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:01.072493 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:01.072501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:01.074706 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:01.074770 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:01.076488 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:01.076496 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:01.076864 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:01.078042 (+ 1178us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:01.078102 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:01.083405 (+ 5303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:01.088813 (+ 5408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:01.088819 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:01.088823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:01.089350 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:01.090054 (+ 704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:01.090059 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:01.090063 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:01.090229 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:01.091203 (+ 974us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:01.091208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:01.091362 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 07:00:01.091501 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 07:00:01.091787 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301}
I20260927 07:00:01.093946 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:01.067380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40144 (local address 127.0.97.60:43633)
0927 07:00:01.067610 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:01.067619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:01.067642 (+ 23us) server_negotiation.cc:407] Connection header received
0927 07:00:01.067742 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:01.067750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:01.067849 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:01.067967 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:01.070773 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:01.072235 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:01.074958 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:01.075908 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:01.076051 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:01.083715 (+ 7664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:01.083723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:01.083728 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:01.083769 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:01.088587 (+ 4818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:01.089575 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:01.089585 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:01.089591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:01.089834 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:01.090409 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:01.090414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:01.090417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:01.091004 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:01.092571 (+ 1567us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:01.092924 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 07:00:01.093332 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 07:00:01.094072 23409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3829c3c7c997488db296e6ac759f7949. Latest consensus state: current_term: 1 leader_uuid: "3829c3c7c997488db296e6ac759f7949" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } } peers { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } } peers { permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } } }
I20260927 07:00:01.094509 23409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [sys.catalog]: This master's current role is: LEADER
I20260927 07:00:01.095669 23417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:00:01.101274 23308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc: Bootstrap starting.
W20260927 07:00:01.104066 23188 tablet.cc:2369] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:00:01.102478 23418 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 07:00:01.108234 23308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:01.109954 23308 log.cc:824] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc: Log is configured to *not* fsync() on all Append() calls
I20260927 07:00:01.123095 23417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:00:01.106869 23167 master_runner.cc:186] Error getting master registration for 127.0.97.61:41081: OK, instance_id { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" instance_seqno: 1790492400343023 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:00:01.131721 23167 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 07:00:01.154711 23308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc: No bootstrap required, opened a new log
I20260927 07:00:01.156879 23357 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36 [term 1 FOLLOWER]: Refusing update from remote peer 3829c3c7c997488db296e6ac759f7949: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:00:01.157902 23409 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:00:01.168515 23308 raft_consensus.cc:348] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } } peers { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } } peers { permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } }
I20260927 07:00:01.169049 23308 raft_consensus.cc:729] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f438704b99b8401c9dd69b02a9bd33dc, State: Initialized, Role: FOLLOWER
I20260927 07:00:01.169652 23308 consensus_queue.cc:260] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } } peers { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } } peers { permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } }
I20260927 07:00:01.171875 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:01.144843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47142 (local address 127.0.97.61:41081)
0927 07:00:01.145048 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:01.145057 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:01.145866 (+ 809us) server_negotiation.cc:407] Connection header received
0927 07:00:01.146405 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:01.146414 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:01.146521 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:01.146614 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:01.151244 (+ 4630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:01.152244 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:01.155737 (+ 3493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:01.156415 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:01.156509 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:01.161709 (+ 5200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:01.161721 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:01.161726 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:01.161768 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:01.165791 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:01.167198 (+ 1407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:01.167205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:01.167208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:01.167468 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:01.168638 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:01.168643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:01.168647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:01.169048 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:01.169296 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:01.170396 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0927 07:00:01.170997 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 07:00:01.186334 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:01.145436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.0.97.62:35653)
0927 07:00:01.145613 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:01.145620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:01.153613 (+ 7993us) server_negotiation.cc:407] Connection header received
0927 07:00:01.153870 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:01.153876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:01.153951 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:01.154072 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:01.166152 (+ 12080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:01.167479 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:01.169813 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:01.170560 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:01.170671 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:01.176837 (+ 6166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:01.176844 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:01.176848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:01.176888 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:01.180955 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:01.182393 (+ 1438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:01.182402 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:01.182408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:01.182628 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:01.183476 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:01.183485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:01.183490 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:01.183974 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:01.184244 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:01.185179 (+ 935us) server_negotiation.cc:299] Negotiation successful
0927 07:00:01.185604 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 07:00:01.193665 23388 sys_catalog.cc:455] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3829c3c7c997488db296e6ac759f7949. Latest consensus state: current_term: 1 leader_uuid: "3829c3c7c997488db296e6ac759f7949" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } } peers { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } } peers { permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } } }
I20260927 07:00:01.194056 23388 sys_catalog.cc:458] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:00:01.202258 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3829c3c7c997488db296e6ac759f7949" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f438704b99b8401c9dd69b02a9bd33dc"
I20260927 07:00:01.202759 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:01.151975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40156 (local address 127.0.97.60:43633)
0927 07:00:01.152160 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:01.152166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:01.173906 (+ 21740us) server_negotiation.cc:407] Connection header received
0927 07:00:01.174338 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:01.174345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:01.174450 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:01.174523 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:01.177730 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:01.179250 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:01.183647 (+ 4397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:01.184636 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:01.184766 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:01.191910 (+ 7144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:01.191921 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:01.191929 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:01.191998 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:01.196608 (+ 4610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:01.198421 (+ 1813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:01.198430 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:01.198437 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:01.198819 (+ 382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:01.199836 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:01.199847 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:01.199852 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:01.200577 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:01.200978 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:01.201903 (+ 925us) server_negotiation.cc:299] Negotiation successful
0927 07:00:01.202399 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 07:00:01.202775 23282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:01.202227 23308 sys_catalog.cc:564] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:00:01.206867 23282 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc [term 1 FOLLOWER]: Refusing update from remote peer 3829c3c7c997488db296e6ac759f7949: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:00:01.207477 23428 sys_catalog.cc:455] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } } peers { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } } peers { permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } } }
I20260927 07:00:01.207963 23428 sys_catalog.cc:458] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:00:01.207850 23409 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:00:01.211401 23283 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3829c3c7c997488db296e6ac759f7949 in term 1.
I20260927 07:00:01.214994 23423 mvcc.cc:205] Tried to move back new op lower bound from 7333856875127619584 to 7333856874846339072. Current Snapshot: MvccSnapshot[applied={T|T < 7333856875127619584}]
I20260927 07:00:01.231460 23409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3829c3c7c997488db296e6ac759f7949" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } } peers { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } } peers { permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } } }
I20260927 07:00:01.231822 23409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [sys.catalog]: This master's current role is: LEADER
I20260927 07:00:01.232821 23417 catalog_manager.cc:1347] Generated new cluster ID: 3756c85954e74dc3be9b745652140cf5
I20260927 07:00:01.232926 23417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:00:01.233764 23388 sys_catalog.cc:455] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3829c3c7c997488db296e6ac759f7949" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } } peers { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } } peers { permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } } }
I20260927 07:00:01.234877 23388 sys_catalog.cc:458] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:00:01.248829 23428 sys_catalog.cc:455] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3829c3c7c997488db296e6ac759f7949. Latest consensus state: current_term: 1 leader_uuid: "3829c3c7c997488db296e6ac759f7949" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } } peers { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } } peers { permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } } }
I20260927 07:00:01.249267 23428 sys_catalog.cc:458] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:00:01.252856 23444 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 07:00:01.253444 23417 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:00:01.254747 23409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3829c3c7c997488db296e6ac759f7949" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } } peers { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } } peers { permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } } }
I20260927 07:00:01.255013 23409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949 [sys.catalog]: This master's current role is: LEADER
I20260927 07:00:01.255265 23417 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:00:01.264612 23428 sys_catalog.cc:455] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3829c3c7c997488db296e6ac759f7949" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3829c3c7c997488db296e6ac759f7949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35653 } } peers { permanent_uuid: "f438704b99b8401c9dd69b02a9bd33dc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41081 } } peers { permanent_uuid: "d962f584390441b2b7cc1d622a629c36" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43633 } } }
I20260927 07:00:01.266902 23428 sys_catalog.cc:458] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:00:01.269521 23417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3829c3c7c997488db296e6ac759f7949: Generated new TSK 0
I20260927 07:00:01.270560 23417 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 07:00:01.346096 23313 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:01.343429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47148 (local address 127.0.97.61:41081)
0927 07:00:01.343614 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:01.343621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:01.344172 (+ 551us) server_negotiation.cc:407] Connection header received
0927 07:00:01.344461 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:01.344469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:01.344567 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:01.344644 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:01.345778 (+ 1134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47148: BlockingRecv error: recv got EOF from 127.0.0.1:47148 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 07:00:01.348381 23236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:01.344303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.0.97.62:35653)
0927 07:00:01.344519 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:01.344526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:01.346707 (+ 2181us) server_negotiation.cc:407] Connection header received
0927 07:00:01.346958 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:01.346963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:01.347064 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:01.347146 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:01.348149 (+ 1003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56896: BlockingRecv error: recv got EOF from 127.0.0.1:56896 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 07:00:01.356176 23385 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:01.353627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.0.97.60:43633)
0927 07:00:01.353804 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:01.353810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:01.354395 (+ 585us) server_negotiation.cc:407] Connection header received
0927 07:00:01.354669 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:01.354675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:01.354769 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:01.354844 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:01.355917 (+ 1073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40172: BlockingRecv error: recv got EOF from 127.0.0.1:40172 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 07:00:01.382467 23313 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:01.379974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.0.97.61:41081)
0927 07:00:01.380179 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:01.380188 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:01.380742 (+ 554us) server_negotiation.cc:407] Connection header received
0927 07:00:01.380981 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:01.380987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:01.381132 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:01.381266 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:01.382205 (+ 939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47154: BlockingRecv error: recv got EOF from 127.0.0.1:47154 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 07:00:01.389213 23313 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:01.387269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.0.97.61:41081)
0927 07:00:01.387440 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:01.387445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:01.387465 (+ 20us) server_negotiation.cc:407] Connection header received
0927 07:00:01.387749 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:01.387756 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:01.387851 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:01.387935 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:01.388988 (+ 1053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47156: BlockingRecv error: recv got EOF from 127.0.0.1:47156 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 07:00:01.397583 23313 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:01.395266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47172 (local address 127.0.97.61:41081)
0927 07:00:01.395455 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:01.395462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:01.395986 (+ 524us) server_negotiation.cc:407] Connection header received
0927 07:00:01.396219 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:01.396226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:01.396334 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:01.396451 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:01.397331 (+ 880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47172: BlockingRecv error: recv got EOF from 127.0.0.1:47172 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 07:00:01.424885 23313 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:01.422809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.0.97.61:41081)
0927 07:00:01.423082 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:01.423088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:01.423299 (+ 211us) server_negotiation.cc:407] Connection header received
0927 07:00:01.423414 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:01.423420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:01.423548 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:01.423667 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:01.424607 (+ 940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47184: BlockingRecv error: recv got EOF from 127.0.0.1:47184 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
W20260927 07:00:01.440798 23313 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:01.438332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47196 (local address 127.0.97.61:41081)
0927 07:00:01.438538 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:01.438544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:01.439073 (+ 529us) server_negotiation.cc:407] Connection header received
0927 07:00:01.439388 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:01.439395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:01.439489 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:01.439562 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:01.440564 (+ 1002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47196: BlockingRecv error: recv got EOF from 127.0.0.1:47196 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 07:00:01.473820 23313 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:01.471645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47200 (local address 127.0.97.61:41081)
0927 07:00:01.471825 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:01.471832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:01.472353 (+ 521us) server_negotiation.cc:407] Connection header received
0927 07:00:01.472538 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:01.472545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:01.472658 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:01.472774 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:01.473572 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47200: BlockingRecv error: recv got EOF from 127.0.0.1:47200 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 07:00:01.511951 23313 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:01.509649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47208 (local address 127.0.97.61:41081)
0927 07:00:01.509843 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:01.509850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:01.510497 (+ 647us) server_negotiation.cc:407] Connection header received
0927 07:00:01.510699 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:01.510705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:01.510800 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:01.510911 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:01.511686 (+ 775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47208: BlockingRecv error: recv got EOF from 127.0.0.1:47208 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 07:00:01.581092 23313 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:01.579225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.0.97.61:41081)
0927 07:00:01.579465 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:01.579471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:01.579756 (+ 285us) server_negotiation.cc:407] Connection header received
0927 07:00:01.579905 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:01.579911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:01.580008 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:01.580141 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:01.580872 (+ 731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47220: BlockingRecv error: recv got EOF from 127.0.0.1:47220 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 07:00:01.714401 23313 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:01.712401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47228 (local address 127.0.97.61:41081)
0927 07:00:01.712640 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:01.712648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:01.712944 (+ 296us) server_negotiation.cc:407] Connection header received
0927 07:00:01.713173 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:01.713179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:01.713258 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:01.713368 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:01.714139 (+ 771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47228: BlockingRecv error: recv got EOF from 127.0.0.1:47228 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 07:00:01.934738 23399 catalog_manager.cc:1259] Loaded cluster ID: 3756c85954e74dc3be9b745652140cf5
I20260927 07:00:01.934839 23399 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36: loading cluster ID for follower catalog manager: success
I20260927 07:00:01.937255 23399 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36: acquiring CA information for follower catalog manager: success
I20260927 07:00:01.939005 23399 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d962f584390441b2b7cc1d622a629c36: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 07:00:01.977190 23313 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:01.975364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.0.97.61:41081)
0927 07:00:01.975572 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:01.975578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:01.975708 (+ 130us) server_negotiation.cc:407] Connection header received
0927 07:00:01.975976 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:01.975981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:01.976053 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:01.976163 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:01.976961 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47234: BlockingRecv error: recv got EOF from 127.0.0.1:47234 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 07:00:02.257030 23444 catalog_manager.cc:1259] Loaded cluster ID: 3756c85954e74dc3be9b745652140cf5
I20260927 07:00:02.257229 23444 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc: loading cluster ID for follower catalog manager: success
I20260927 07:00:02.259729 23444 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc: acquiring CA information for follower catalog manager: success
I20260927 07:00:02.261385 23444 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f438704b99b8401c9dd69b02a9bd33dc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:00:02.507500 23511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:02.499904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.0.97.61:41081)
0927 07:00:02.500542 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:02.500548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:02.500883 (+ 335us) server_negotiation.cc:407] Connection header received
0927 07:00:02.501134 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:02.501138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:02.501289 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:02.501390 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:00:02.501957 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:02.502860 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:02.504807 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:02.505297 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:02.505426 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:02.506191 (+ 765us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:00:02.506800 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:00:02.506904 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:02.507080 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 07:00:02.507160 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":301,"threads_started":1}
I20260927 07:00:02.519853 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:02.513556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40174 (local address 127.0.97.60:43633)
0927 07:00:02.514180 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:02.514185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:02.514202 (+ 17us) server_negotiation.cc:407] Connection header received
0927 07:00:02.514288 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:02.514291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:02.514424 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:02.514553 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:00:02.514992 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:02.515634 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:02.517610 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:02.518129 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:02.518260 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:02.518794 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:00:02.519218 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:00:02.519324 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:02.519488 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 07:00:02.519551 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":394,"threads_started":1}
I20260927 07:00:02.546886 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:02.526101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56910 (local address 127.0.97.62:35653)
0927 07:00:02.538782 (+ 12681us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:02.538791 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:02.538816 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:00:02.538956 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:02.538964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:02.539206 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:02.539393 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:00:02.540450 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:02.541628 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:02.544062 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:02.544722 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:02.545281 (+ 559us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:02.545456 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:00:02.545880 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:00:02.546399 (+ 519us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:02.546451 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 07:00:02.546541 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12522,"thread_start_us":14180,"threads_started":1}
I20260927 07:00:02.547597 23511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:02.526501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.0.97.61:41081)
0927 07:00:02.532369 (+ 5868us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:02.532377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:02.540129 (+ 7752us) server_negotiation.cc:407] Connection header received
0927 07:00:02.540331 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:02.540336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:02.540421 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:02.540525 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:00:02.541305 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:02.542507 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:02.544892 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:02.545575 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:02.545671 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:02.546336 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:00:02.546454 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:00:02.546535 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:02.547225 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 07:00:02.547303 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5717}
I20260927 07:00:02.547976 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:02.533671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.0.97.60:43633)
0927 07:00:02.533860 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:02.533868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:02.541305 (+ 7437us) server_negotiation.cc:407] Connection header received
0927 07:00:02.541646 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:02.541652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:02.541752 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:02.541809 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:00:02.542721 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:02.543789 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:02.545790 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:02.546431 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:02.546515 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:02.547239 (+ 724us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:00:02.547367 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:00:02.547430 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:02.547643 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 07:00:02.547718 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 07:00:02.551760 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 23167
I20260927 07:00:02.552811 23229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:00:02.659781 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 23167
W20260927 07:00:02.679507 23333 connection.cc:535] server connection from 127.0.0.1:40128 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 07:00:02.679913 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 23241
I20260927 07:00:02.680898 23303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:00:02.782852 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 23241
I20260927 07:00:02.802196 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu pid 23315
I20260927 07:00:02.803285 23377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:00:02.911715 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktaM5HI/build/asan/bin/kudu with pid 23315
2026-09-27T07:00:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4778 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9725 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (355403 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST